From 7b429620aaab818a0bbfaf14a694b0b3479f47e8 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 15 Oct 2026 21:01:11 +0300 Subject: [PATCH 01/44] fix: restore the declarative monitoring evidence contract Label external monitoring plans in manifests and Annex IV, warn on inert exporter declarations, and preserve unresolved endpoint references with explicit precedence and null handling. Align CLI/web wizards, six site locales, EN/TR references and QMS guidance; add regression coverage. Resolves CR-A01-007, TECH-T09-003, DOC-D1B01-004, DOC-D3B04A-002, DOC-BLEGAL-048, DOC-D1B05-003, DOC-D4B02-003, DOC-D4B09-008, DOC-D4B09-009, DOC-BFIELD-017, DOC-D3B04A-014, DOC-BFIELD-031. Fixes #93 Co-authored-by: Codex --- CHANGELOG.md | 6 + config_template.yaml | 6 +- docs/qms/sop_incident_response-tr.md | 3 +- docs/qms/sop_incident_response.md | 3 +- docs/reference/configuration-tr.md | 25 ++- docs/reference/configuration.md | 25 ++- .../verify_annex_iv_subcommand-tr.md | 2 +- docs/reference/verify_annex_iv_subcommand.md | 2 +- docs/usermanuals/en/compliance/annex-iv.md | 2 +- docs/usermanuals/en/compliance/overview.md | 2 +- .../usermanuals/en/reference/configuration.md | 24 ++- docs/usermanuals/tr/compliance/annex-iv.md | 2 +- docs/usermanuals/tr/compliance/overview.md | 2 +- .../usermanuals/tr/reference/configuration.md | 24 ++- forgelm/compliance.py | 7 +- forgelm/config.py | 32 +++- forgelm/wizard/_collectors.py | 17 +- site/js/translations.js | 162 +++++++++--------- site/js/wizard.js | 42 +---- tests/test_config.py | 25 +++ tests/test_eu_ai_act.py | 55 ++++++ tests/test_wizard_phase22.py | 9 + tools/check_module_size.py | 13 +- 23 files changed, 312 insertions(+), 178 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index febf460c..0d620f6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,12 @@ All notable changes to ForgeLM are documented here. ### Fixed +- Restore the monitoring-plan evidence contract: `monitoring.*` declares an external plan, + labelled `declarative_only: true` in the compliance manifest and Annex IV; ForgeLM runs no + exporter, drift detector, alert, or scheduler. Warn when an enabled plan names an exporter, + record absent endpoints as `null`, and honor unresolved `endpoint_env` references over literal + endpoints. Align CLI/web wizards, six site locales, and EN/TR references and QMS guidance; + correct the site's claim that webhook delivery attempts are recorded in the audit log (#93). - Rework the homepage terminal into a first-run guide with separately copyable installation and config-generation commands, an optional training step, and readable code in both themes and on narrow screens, with interface copy in all six site languages. Keep the dated CPU recording and diff --git a/config_template.yaml b/config_template.yaml index ce08c2d6..50e7b499 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -256,10 +256,12 @@ data: # - "Human review required for policy-related responses" # vulnerable_groups_considered: true -# Post-Market Monitoring (Art. 12+17) +# External post-market monitoring plan (Art. 72 / 17(1)(h)) — declarative metadata only. +# Recorded in the compliance manifest / Annex IV; ForgeLM does not contact the endpoint, +# export metrics, detect drift, send drift alerts, or schedule checks. Operate your own stack. # monitoring: # enabled: true -# endpoint_env: "MONITORING_WEBHOOK_URL" +# endpoint_env: "MONITORING_WEBHOOK_URL" # recorded as "$NAME", never read; overrides endpoint # metrics_export: "prometheus" # "none", "prometheus", "datadog", "custom_webhook" # alert_on_drift: true # check_interval_hours: 24 diff --git a/docs/qms/sop_incident_response-tr.md b/docs/qms/sop_incident_response-tr.md index d178022d..56713391 100644 --- a/docs/qms/sop_incident_response-tr.md +++ b/docs/qms/sop_incident_response-tr.md @@ -28,7 +28,8 @@ yanında kapsar. ### 3.1 Tespit Olaylar şunlardan tespit edilebilir: -- Runtime izleme alarmları (`monitoring.alert_on_drift: true` ise) +- Harici servis/izleme altyapınızdan gelen runtime uyarıları (`monitoring.*` uyumluluk manifestine + yalnızca bir plan kaydeder; ForgeLM model sapması tespit etmez veya izleme uyarısı göndermez) - Kullanıcı/operatör raporları - Periyodik kalite denetimleri - ForgeLM webhook arıza bildirimleri diff --git a/docs/qms/sop_incident_response.md b/docs/qms/sop_incident_response.md index 4ccd2aff..bf322545 100644 --- a/docs/qms/sop_incident_response.md +++ b/docs/qms/sop_incident_response.md @@ -28,7 +28,8 @@ existing AI-safety incident flow. ### 3.1 Detection Incidents may be detected by: -- Runtime monitoring alerts (if `monitoring.alert_on_drift: true`) +- Runtime alerts from your external serving/monitoring stack (`monitoring.*` only records a plan + in the compliance manifest; ForgeLM does not detect drift or send monitoring alerts) - User/deployer reports - Periodic quality audits - ForgeLM webhook failure notifications diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index bc7c49ef..689add6d 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -316,16 +316,27 @@ uzatmasını engeller. > **Sıkı kapı:** yukarıdaki [`compliance.risk_classification`](#compliance-isteğe-bağlı--eu-ai-act-madde-11--annex-iv) ile aynı — `risk_category`'i `"high-risk"` veya `"unacceptable"` olarak ayarlamak `evaluation.safety.enabled: true` gerektirir, aksi halde config-load `ConfigError` (çıkış kodu 1) fırlatır. Kapı her iki alan üzerinden OR'lanır: ikisinden biri sıkı bir tier'daysa kapı tetiklenir. -## `monitoring` (İsteğe bağlı — EU AI Act Madde 12+17) +## `monitoring` (İsteğe bağlı — Harici izleme planı, Madde 72 / 17(1)(h)) + +Bu blok **yalnızca beyan niteliğinde metadata** içerir. `enabled: true` olduğunda ForgeLM, harici +pazar sonrası izleme planınızı uyumluluk manifestine ve Annex IV `monitoring_and_logging` bölümüne +`declarative_only: true` etiketiyle kaydeder. Endpoint'e bağlanmaz, metrik aktarmaz, model sapması +tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. İzleme altyapınızı ayrıca işletin; +bu beyan izlemenin çalıştığını kanıtlamaz ve izleme yükümlülüğünü tek başına karşılamaz. +Pazar sonrası izleme planı [Madde 72](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-72), +QMS ise [Madde 17(1)(h)](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-17) kapsamındadır. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `enabled` | bool | `false` | İzleme hook'larını etkinleştir | -| `endpoint` | string | `""` | İzleme webhook URL'si | -| `endpoint_env` | string | `null` | Endpoint için ortam değişkeni adı | -| `metrics_export` | string | `"none"` | `"none"`, `"prometheus"`, `"datadog"`, `"custom_webhook"` | -| `alert_on_drift` | bool | `true` | Model sapmasında uyar | -| `check_interval_hours` | int | `24` | İzleme kontrol aralığı (saat) | +| `enabled` | bool | `false` | Harici planı manifeste kaydet; tüm risk seviyelerinde isteğe bağlıdır | +| `endpoint` | string | `""` | Beyan edilen harici izleme URL'si; isteğe bağlıdır, bağlantı kurulmaz | +| `endpoint_env` | string | `null` | Boş olmayan değişken adı `endpoint`'ten önceliklidir; `$NAME` olarak kaydedilir, değeri okunmaz | +| `metrics_export` | string | `"none"` | Beyan edilen harici aktarım sistemi: `"none"`, `"prometheus"`, `"datadog"`, `"custom_webhook"`; etkin planda `none` dışındaki seçim yapılandırma yüklenirken uyarı üretir | +| `alert_on_drift` | bool | `true` | Harici planda sapma uyarısı öngörülüp öngörülmediği; uyarıları etkinleştirmez | +| `check_interval_hours` | int | `24` | Beyan edilen harici kontrol aralığı, saat (en az 1); kontrol zamanlamaz | + +Her iki endpoint alanı da boşsa manifeste `endpoint: null` yazılır. Adlandırılan ortam değişkeni +tanımlı olsa bile referansı çözümlenmez. Kapalı veya belirtilmemiş planlar kaydedilmez. ## `distributed` (İsteğe bağlı) diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 69f04b76..cf52cc69 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -374,16 +374,27 @@ silently extend the retention horizon by re-using a stale workspace. --- -## `monitoring` (Optional — EU AI Act Art. 12+17) +## `monitoring` (Optional — External monitoring plan, Art. 72 / 17(1)(h)) + +This block is **declarative metadata only**. With `enabled: true`, ForgeLM records your external +post-market monitoring plan in the compliance manifest and Annex IV `monitoring_and_logging` section, +labelled `declarative_only: true`. It does not contact the endpoint, export metrics, detect drift, +send drift alerts, or schedule checks. Operate your monitoring stack separately; this declaration +does not establish that monitoring is running or satisfy the monitoring obligation by itself. +The post-market monitoring plan is covered by [Article 72](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-72) +and the QMS by [Article 17(1)(h)](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-17). | Field | Type | Default | Description | |-------|------|---------|-------------| -| `enabled` | bool | `false` | Enable monitoring hooks | -| `endpoint` | string | `""` | Monitoring webhook URL | -| `endpoint_env` | string | `null` | Env var name for endpoint | -| `metrics_export` | string | `"none"` | `"none"`, `"prometheus"`, `"datadog"`, `"custom_webhook"` | -| `alert_on_drift` | bool | `true` | Alert on model drift | -| `check_interval_hours` | int | `24` | Monitoring check interval | +| `enabled` | bool | `false` | Record the external plan in the manifest; opt-in for every risk tier | +| `endpoint` | string | `""` | Declared external monitoring URL; optional, never contacted | +| `endpoint_env` | string | `null` | Non-empty variable name takes precedence over `endpoint`; recorded as `$NAME`, never read | +| `metrics_export` | string | `"none"` | Declared external exporter: `"none"`, `"prometheus"`, `"datadog"`, `"custom_webhook"`; an enabled non-`none` declaration emits a config-load warning | +| `alert_on_drift` | bool | `true` | Whether the external plan calls for drift alerts; does not activate alerts | +| `check_interval_hours` | int | `24` | Declared external cadence in hours (minimum 1); does not schedule checks | + +If both endpoint fields are empty, the manifest records `endpoint: null`. Environment-variable +references remain unresolved even if the named variable is set. Disabled or omitted plans are not recorded. --- diff --git a/docs/reference/verify_annex_iv_subcommand-tr.md b/docs/reference/verify_annex_iv_subcommand-tr.md index 699d0a41..ae5b7098 100644 --- a/docs/reference/verify_annex_iv_subcommand-tr.md +++ b/docs/reference/verify_annex_iv_subcommand-tr.md @@ -166,7 +166,7 @@ Doğrulayıcı statik bir katalogu (`_ANNEX_IV_REQUIRED_FIELDS`) yürür; böyle | `computational_resources` | §2(g) — eğitim sırasında kullanılan hesaplama kaynakları. | | `data_governance` | §2(d) — veri kaynakları, yönetişim, doğrulama metodolojisi. | | `technical_documentation` | §3-5 — tasarım + geliştirme metodolojisi. | -| `monitoring_and_logging` | §6 — pazara-sonrası izleme + audit-log varlığı. | +| `monitoring_and_logging` | Beyan edilen harici izleme planı (`declarative_only: true`) veya audit log referansı; çalışan izleme kanıtı değildir. | | `performance_metrics` | §7 — doğruluk / dayanıklılık / siber güvenlik metrikleri. | | `risk_management` | §9 — risk yönetim sistemi referansı (Madde 9 hizalaması). | diff --git a/docs/reference/verify_annex_iv_subcommand.md b/docs/reference/verify_annex_iv_subcommand.md index c8fdc943..6a9d8397 100644 --- a/docs/reference/verify_annex_iv_subcommand.md +++ b/docs/reference/verify_annex_iv_subcommand.md @@ -166,7 +166,7 @@ The verifier walks a static catalog (`_ANNEX_IV_REQUIRED_FIELDS`) so a future sc | `computational_resources` | §2(g) — compute resources used during training. | | `data_governance` | §2(d) — data sources, governance, validation methodology. | | `technical_documentation` | §3-5 — design + development methodology. | -| `monitoring_and_logging` | §6 — post-market monitoring + audit-log presence. | +| `monitoring_and_logging` | Declared external monitoring-plan metadata (`declarative_only: true`), or an audit-log reference; no runtime monitoring evidence. | | `performance_metrics` | §7 — accuracy / robustness / cybersecurity metrics. | | `risk_management` | §9 — risk management system reference (Article 9 alignment). | diff --git a/docs/usermanuals/en/compliance/annex-iv.md b/docs/usermanuals/en/compliance/annex-iv.md index a2aa31ce..cdd8746a 100644 --- a/docs/usermanuals/en/compliance/annex-iv.md +++ b/docs/usermanuals/en/compliance/annex-iv.md @@ -82,7 +82,7 @@ The nine required keys map onto Annex IV §1-9: | `computational_resources` | §2(g) — compute resources used during training. | | `data_governance` | §2(d) — data sources, governance, validation methodology. | | `technical_documentation` | §3-5 — design + development methodology. | -| `monitoring_and_logging` | §6 — post-market monitoring + audit-log presence. | +| `monitoring_and_logging` | Declared external monitoring-plan metadata (`declarative_only: true`), or an audit-log reference; no runtime monitoring evidence. | | `performance_metrics` | §7 — accuracy / robustness / cybersecurity metrics. | | `risk_management` | §9 — risk management system reference (Article 9 alignment). | diff --git a/docs/usermanuals/en/compliance/overview.md b/docs/usermanuals/en/compliance/overview.md index 1a800083..92b616b3 100644 --- a/docs/usermanuals/en/compliance/overview.md +++ b/docs/usermanuals/en/compliance/overview.md @@ -132,7 +132,7 @@ The artifact carries **nine** top-level keys mapped onto Annex IV §1-9, populat | `computational_resources` | §2(g) — compute used during training. | | `data_governance` | §2(d) — data provenance and validation methodology. | | `technical_documentation` | §3-5 — ForgeLM version, generation timestamp, known limitations. | -| `monitoring_and_logging` | §6 — post-market monitoring + audit-log reference. | +| `monitoring_and_logging` | Declared external monitoring-plan metadata (`declarative_only: true`), or an audit-log reference; no runtime monitoring evidence. | | `performance_metrics` | §7 — accuracy / robustness metrics from the evaluation results. | | `risk_management` | §9 — the top-level `risk_assessment:` block, or an explicit "no risk_assessment block configured" marker. | diff --git a/docs/usermanuals/en/reference/configuration.md b/docs/usermanuals/en/reference/configuration.md index 280bf944..e52296e4 100644 --- a/docs/usermanuals/en/reference/configuration.md +++ b/docs/usermanuals/en/reference/configuration.md @@ -200,7 +200,7 @@ synthetic: sanity_failure_rate: 0.2 # failure rate above which a WARNING is logged (warn-only) ``` -There is no nested `synthetic.teacher:` sub-block and no `rate_limit:` block — `teacher_model`, `teacher_backend`, `api_base`, `api_delay`, and `api_timeout` are flat fields on `synthetic:` directly. Prefer `api_key_env` (an environment-variable name) over the inline `api_key` field to avoid committing secrets. ForgeLM's YAML loader has no `${VAR}` interpolation mechanism anywhere — `*_env` fields name an environment variable that is read directly via `os.environ`, never substituted into a string. +There is no nested `synthetic.teacher:` sub-block and no `rate_limit:` block — `teacher_model`, `teacher_backend`, `api_base`, `api_delay`, and `api_timeout` are flat fields on `synthetic:` directly. Prefer `api_key_env` (an environment-variable name) over the inline `api_key` field to avoid committing secrets. ForgeLM's YAML loader has no `${VAR}` interpolation mechanism anywhere — `*_env` fields name an environment variable that is read directly via `os.environ`, never substituted into a string, except `monitoring.endpoint_env`: that name is recorded as an unresolved `$NAME` reference only. Under `teacher_backend: "local"` you may also set `teacher_revision` to pin the teacher checkpoint to a Hub commit SHA or ref; the teacher's generations become training data, so an unpinned teacher is a data-provenance gap. It is rejected outright with `teacher_backend: "api"` or `"file"`, which never load from the Hub. @@ -286,16 +286,26 @@ risk_assessment: ## `monitoring:` +This block declares an external post-market monitoring plan (Art. 72 / 17(1)(h)). With `enabled: true`, +its values are recorded in the compliance manifest and Annex IV `monitoring_and_logging` section with +`declarative_only: true`. ForgeLM does not contact the endpoint, export metrics, detect drift, send +drift alerts, or schedule checks. Operate your monitoring stack separately; the declaration is not +proof that monitoring is running. The block remains optional for every risk tier. + ```yaml monitoring: - enabled: false # Enable Article 12 post-market monitoring - endpoint: "" # Monitoring webhook URL (Prometheus / Datadog / custom) - endpoint_env: null # env var overriding endpoint - metrics_export: "none" # none | prometheus | datadog | custom_webhook - alert_on_drift: true # webhook alert on drift-detector regression - check_interval_hours: 24 # monitoring cadence in hours + enabled: false # Record external plan metadata only + endpoint: "" # Optional external URL; never contacted + endpoint_env: null # Non-empty name overrides endpoint as "$NAME"; never read + metrics_export: "none" # Declared only: none | prometheus | datadog | custom_webhook + alert_on_drift: true # Declared external drift-alert policy; no alerts sent + check_interval_hours: 24 # Declared external cadence; no checks scheduled ``` +An enabled non-`none` exporter declaration emits a config-load warning. If neither endpoint field +is set, the manifest endpoint is `null`. Environment references are recorded by name even when the +variable is set; disabled or omitted plans are not recorded. + ## `retention:` ```yaml diff --git a/docs/usermanuals/tr/compliance/annex-iv.md b/docs/usermanuals/tr/compliance/annex-iv.md index 76560ba4..5e357887 100644 --- a/docs/usermanuals/tr/compliance/annex-iv.md +++ b/docs/usermanuals/tr/compliance/annex-iv.md @@ -82,7 +82,7 @@ Dokuz gerekli anahtar Annex IV §1-9 ile eşleşir: | `computational_resources` | §2(g) — eğitim sırasında kullanılan hesaplama kaynakları. | | `data_governance` | §2(d) — veri kaynakları, yönetişim, doğrulama metodolojisi. | | `technical_documentation` | §3-5 — tasarım + geliştirme metodolojisi. | -| `monitoring_and_logging` | §6 — pazara-sonrası izleme + audit-log varlığı. | +| `monitoring_and_logging` | Beyan edilen harici izleme planı (`declarative_only: true`) veya audit log referansı; çalışan izleme kanıtı değildir. | | `performance_metrics` | §7 — doğruluk / dayanıklılık / siber güvenlik metrikleri. | | `risk_management` | §9 — risk yönetim sistemi referansı (Madde 9 hizalaması). | diff --git a/docs/usermanuals/tr/compliance/overview.md b/docs/usermanuals/tr/compliance/overview.md index 36a6c4a4..6350fd0b 100644 --- a/docs/usermanuals/tr/compliance/overview.md +++ b/docs/usermanuals/tr/compliance/overview.md @@ -132,7 +132,7 @@ Artefact, eğitim manifestinden ve `compliance:` bloğunuzdan doldurulan, Annex | `computational_resources` | §2(g) — eğitim sırasında kullanılan hesaplama kaynağı. | | `data_governance` | §2(d) — veri kökeni ve doğrulama metodolojisi. | | `technical_documentation` | §3-5 — ForgeLM sürümü, üretim zaman damgası, bilinen sınırlamalar. | -| `monitoring_and_logging` | §6 — pazar-sonrası izleme + audit log referansı. | +| `monitoring_and_logging` | Beyan edilen harici izleme planı (`declarative_only: true`) veya audit log referansı; çalışan izleme kanıtı değildir. | | `performance_metrics` | §7 — değerlendirme sonuçlarından doğruluk / sağlamlık metrikleri. | | `risk_management` | §9 — top-level `risk_assessment:` bloğu ya da açık bir "no risk_assessment block configured" işareti. | diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index f45afc40..6d7e3923 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -202,7 +202,7 @@ synthetic: sanity_failure_rate: 0.2 # üzerinde bir WARNING loglanan başarısızlık oranı (yalnızca uyarı) ``` -Nested `synthetic.teacher:` alt-bloğu ve `rate_limit:` bloğu yoktur — `teacher_model`, `teacher_backend`, `api_base`, `api_delay`, `api_timeout` doğrudan `synthetic:` üzerinde düz alanlardır. Secret'ları commit etmemek için inline `api_key` alanı yerine `api_key_env` (bir ortam-değişkeni adı) tercih edin. ForgeLM'in YAML loader'ında hiçbir yerde `${VAR}` interpolasyon mekanizması yoktur — `*_env` alanları doğrudan `os.environ` ile okunan bir ortam değişkeni adlandırır, bir string içine substitüe edilmez. +Nested `synthetic.teacher:` alt-bloğu ve `rate_limit:` bloğu yoktur — `teacher_model`, `teacher_backend`, `api_base`, `api_delay`, `api_timeout` doğrudan `synthetic:` üzerinde düz alanlardır. Secret'ları commit etmemek için inline `api_key` alanı yerine `api_key_env` (bir ortam-değişkeni adı) tercih edin. ForgeLM'in YAML loader'ında hiçbir yerde `${VAR}` interpolasyon mekanizması yoktur — `*_env` alanları doğrudan `os.environ` ile okunan bir ortam değişkeni adlandırır, bir string içine substitüe edilmez; istisna `monitoring.endpoint_env` alanıdır: bu ad yalnızca çözümlenmemiş bir `$NAME` referansı olarak kaydedilir. `teacher_backend: "local"` altında ayrıca `teacher_revision` ayarlayarak teacher checkpoint'ini bir Hub commit SHA'sına veya ref'ine sabitleyebilirsiniz; teacher'ın ürettikleri eğitim verisi hâline gelir, dolayısıyla sabitlenmemiş bir teacher bir veri-kökeni boşluğudur. `teacher_backend: "api"` veya `"file"` ile — ki bunlar Hub'dan hiç yüklemez — doğrudan reddedilir. @@ -288,16 +288,26 @@ risk_assessment: ## `monitoring:` +Bu blok harici bir pazar sonrası izleme planını beyan eder (Madde 72 / 17(1)(h)). `enabled: true` +olduğunda değerleri uyumluluk manifestine ve Annex IV `monitoring_and_logging` bölümüne +`declarative_only: true` etiketiyle kaydedilir. ForgeLM endpoint'e bağlanmaz, metrik aktarmaz, +model sapması tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. İzleme altyapınızı ayrıca +işletin; bu beyan izlemenin çalıştığını kanıtlamaz. Blok tüm risk seviyelerinde isteğe bağlıdır. + ```yaml monitoring: - enabled: false # Madde 12 pazar-sonrası izlemeyi etkinleştir - endpoint: "" # İzleme webhook URL'si (Prometheus / Datadog / özel) - endpoint_env: null # endpoint'i geçersiz kılan env değişkeni - metrics_export: "none" # none | prometheus | datadog | custom_webhook - alert_on_drift: true # drift tespitinde webhook uyarısı - check_interval_hours: 24 # izleme periyodu (saat) + enabled: false # Record external plan metadata only + endpoint: "" # Optional external URL; never contacted + endpoint_env: null # Non-empty name overrides endpoint as "$NAME"; never read + metrics_export: "none" # Declared only: none | prometheus | datadog | custom_webhook + alert_on_drift: true # Declared external drift-alert policy; no alerts sent + check_interval_hours: 24 # Declared external cadence; no checks scheduled ``` +Etkin planda `none` dışındaki aktarım sistemi seçimi yapılandırma yüklenirken uyarı üretir. Her iki +endpoint alanı da boşsa manifeste `null` yazılır. Ortam değişkeni tanımlı olsa bile yalnızca adı +kaydedilir; kapalı veya belirtilmemiş planlar kaydedilmez. + ## `retention:` ```yaml diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 9bd94b2a..4861decf 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -1578,11 +1578,12 @@ def generate_training_manifest( {"path": p, "fingerprint": compute_dataset_fingerprint(p, offline=_offline)} for p in extra_datasets ] - # Monitoring config + # Record the operator's plan, not evidence of an active monitoring system. mon_cfg = getattr(config, "monitoring", None) if mon_cfg and mon_cfg.enabled: manifest["monitoring"] = { - "endpoint": mon_cfg.endpoint or f"${mon_cfg.endpoint_env}", + "declarative_only": True, + "endpoint": f"${mon_cfg.endpoint_env}" if mon_cfg.endpoint_env else (mon_cfg.endpoint or None), "metrics_export": mon_cfg.metrics_export, "alert_on_drift": mon_cfg.alert_on_drift, "check_interval_hours": mon_cfg.check_interval_hours, @@ -1853,7 +1854,7 @@ def build_annex_iv_artifact(manifest: Dict[str, Any]) -> Optional[Dict[str, Any] "generated_at": manifest.get("generated_at", ""), "known_limitations": operator_block.get("known_limitations", ""), }, - # Annex IV §6: post-market monitoring + audit-log presence. + # Declared external monitoring metadata, or the audit-log reference. "monitoring_and_logging": (manifest.get("monitoring") or {"audit_log": "audit_log.jsonl"}), # Annex IV §7: accuracy / robustness metrics. "performance_metrics": manifest.get("evaluation_results", {}).get("metrics", {}), diff --git a/forgelm/config.py b/forgelm/config.py index a098f3c8..9e601d48 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1288,25 +1288,41 @@ class RiskAssessmentConfig(BaseModel): class MonitoringConfig(BaseModel): - """Art. 12+17: Post-market monitoring hooks.""" + """Art. 72 / 17(1)(h): declarative external monitoring plan, without runtime hooks.""" model_config = ConfigDict(extra="forbid") - enabled: bool = Field(default=False, description="Enable Article 12 post-market monitoring hooks.") + enabled: bool = Field( + default=False, description="Record external monitoring-plan metadata in the compliance manifest." + ) endpoint: str = Field( - default="", description="Monitoring-system webhook URL (Prometheus push gateway / Datadog / custom)." + default="", description="Declared external monitoring URL; recorded only, never contacted by ForgeLM." ) endpoint_env: Optional[str] = Field( - default=None, description="Env var name carrying the endpoint URL (overrides `endpoint` when set)." + default=None, + description="Env var name recorded as `$NAME`, overriding `endpoint` when non-empty; never resolved.", ) metrics_export: Literal["none", "prometheus", "datadog", "custom_webhook"] = Field( default="none", - description="Metrics exporter: `none`, `prometheus`, `datadog`, or `custom_webhook`.", + description="Declared external exporter: `none`, `prometheus`, `datadog`, or `custom_webhook`; no metrics are exported.", ) alert_on_drift: bool = Field( - default=True, description="Emit a webhook alert when drift detector flags a regression." + default=True, + description="Declared external drift-alert policy; ForgeLM does not detect drift or send these alerts.", + ) + check_interval_hours: int = Field( + default=24, ge=1, description="Declared external check cadence in hours; ForgeLM does not schedule checks." ) - check_interval_hours: int = Field(default=24, ge=1, description="Monitoring check cadence in hours.") + + @model_validator(mode="after") + def _warn_declarative_exporter(self) -> "MonitoringConfig": + if self.enabled and self.metrics_export != "none": + logger.warning( + "monitoring.metrics_export is declarative metadata only: ForgeLM does not export metrics, " + "contact the endpoint, detect drift, send drift alerts, or schedule checks. " + "Operate your external monitoring stack separately." + ) + return self class ComplianceMetadataConfig(BaseModel): @@ -1642,7 +1658,7 @@ class ForgeConfig(BaseModel): description="EU AI Act Article 9 risk-management block (intended use, foreseeable misuse, risk tier).", ) monitoring: Optional[MonitoringConfig] = Field( - default=None, description="EU AI Act Article 12 + 17 post-market monitoring block." + default=None, description="EU AI Act Art. 72 / 17(1)(h) external monitoring-plan metadata; no runtime hooks." ) synthetic: Optional[SyntheticConfig] = Field( default=None, description="Teacher→student synthetic-data generation block." diff --git a/forgelm/wizard/_collectors.py b/forgelm/wizard/_collectors.py index 50f0a6c9..f1dfee6b 100644 --- a/forgelm/wizard/_collectors.py +++ b/forgelm/wizard/_collectors.py @@ -879,28 +879,27 @@ def _collect_retention() -> Optional[Dict[str, Any]]: def _collect_monitoring() -> Optional[Dict[str, Any]]: - """Article 12+17: post-market monitoring hooks. + """Collect declarative external monitoring metadata for the compliance manifest. - Endpoint accepts either a literal URL or ``env:VAR_NAME`` to source - the URL from the environment — mirrors the webhook collector's - convention so the same syntax works in both places. + Endpoint accepts a literal URL or ``env:VAR_NAME`` recorded by name only. """ - if not _prompt_yes_no("Configure post-market monitoring?", default=False): + if not _prompt_yes_no("Record an external post-market monitoring plan (metadata only)?", default=False): return None + _print(" Metadata only: ForgeLM does not export metrics, detect drift, send drift alerts, or schedule checks.") endpoint_raw = _prompt( - "Monitoring endpoint URL (or `env:VAR_NAME` to source from environment)", + "Declared monitoring URL (optional; `env:VAR_NAME` is recorded without reading the variable)", "", ) metrics_export = _prompt_choice( - "metrics_export:", + "Declared external metrics_export:", ["none", "prometheus", "datadog", "custom_webhook"], default=1, ) monitoring: Dict[str, Any] = { "enabled": True, "metrics_export": metrics_export, - "alert_on_drift": _prompt_yes_no("alert_on_drift?", default=True), - "check_interval_hours": _prompt_int("check_interval_hours", 24, min_val=1, max_val=720), + "alert_on_drift": _prompt_yes_no("External monitoring plan: alert_on_drift?", default=True), + "check_interval_hours": _prompt_int("Declared external check_interval_hours", 24, min_val=1, max_val=720), } # P9: support env:VAR_NAME indirection for the monitoring endpoint # the same way the webhook collector does. ``MonitoringConfig`` has diff --git a/site/js/translations.js b/site/js/translations.js index cc00c511..3017a2b9 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -1678,7 +1678,7 @@ "wizard.step.operations.title": "Operations & advanced settings", "wizard.step.operations.name": "Operations", "wizard.step.operations.desc": "Notifications, post-market monitoring, synthetic data generation, and retention horizons. All optional — skip the ones you don't need.", - "wizard.step.operations.tutorial": "These four blocks cover everything ForgeLM ships beyond the core training + compliance flow. Webhook posts a Slack/Teams/Discord alert when training starts, succeeds, or fails. Post-market monitoring (Article 12+17) wires drift detection into Prometheus / Datadog after deployment. Synthetic data generation lets a stronger teacher model produce extra training examples. Retention horizons set GDPR-compliant TTLs for the audit log and intermediate artefacts. None of these are required — leave them collapsed if your run doesn't need them.", + "wizard.step.operations.tutorial": "These four blocks cover everything ForgeLM ships beyond the core training + compliance flow. Webhook posts a Slack/Teams/Discord alert when training starts, succeeds, or fails. Monitoring (Art. 72 / 17(1)(h)) records an external plan as metadata; ForgeLM does not export metrics, detect drift, send drift alerts, or schedule checks. Synthetic data generation lets a stronger teacher model produce extra training examples. Retention horizons set GDPR-compliant TTLs for the audit log and intermediate artefacts. None of these are required — leave them collapsed if your run doesn't need them.", "wizard.risk.label": "EU AI Act risk classification", "wizard.risk.hint": "Pick the tier your deployment falls under. High-risk and unacceptable trigger mandatory audit + human-approval obligations.", "wizard.risk.unknown": "Unknown — pre-classification placeholder", @@ -1775,25 +1775,25 @@ "wizard.webhook.url.detail": "You can paste the URL directly (fine for non-secret test channels) or use the env: prefix to point at an environment variable. The env-var route keeps the URL out of YAML and out of the audit log — required for production secrets.", "wizard.webhook.enabled.label": "Send webhook notifications", "wizard.webhook.enabled.hint": "When on, ForgeLM posts a webhook payload at the events you select. Off keeps the YAML free of any webhook block.", - "wizard.webhook.enabled.detail": "Notifications wire training events into your team chat (Slack / Teams / Discord) or any HTTPS endpoint that accepts JSON. Audit log entries record every dispatch — including failures — so you have an independent trail of what was sent. Leave off if you don't want the YAML to carry webhook config at all.", + "wizard.webhook.enabled.detail": "Notifications wire training events into your team chat (Slack / Teams / Discord) or any HTTPS endpoint that accepts JSON. Delivery failures are warnings in the process log only. The audit log records the underlying training events, not delivery attempts. Leave off if you don't want the YAML to carry webhook config at all.", "wizard.webhook.notify_start.label": "Notify on training start", "wizard.webhook.notify_success.label": "Notify on success", "wizard.webhook.notify_failure.label": "Notify on failure", - "wizard.monitoring.title": "Post-market monitoring (Art. 12+17)", - "wizard.monitoring.hint": "Wire training metrics into Prometheus / Datadog / a custom webhook for ongoing observability.", - "wizard.monitoring.enabled.label": "Enable monitoring hooks", - "wizard.monitoring.enabled.hint": "Required for high-risk systems under EU AI Act Articles 12 + 17.", - "wizard.monitoring.enabled.detail": "Once a high-risk system is deployed, the EU AI Act requires ongoing observation — drift detection, error reporting, periodic accuracy re-checks. ForgeLM doesn't run the monitoring itself, but it wires the right hooks into the training run so your existing observability stack picks them up.", - "wizard.monitoring.endpoint.label": "Monitoring endpoint", - "wizard.monitoring.endpoint.hint": "URL of your Prometheus push gateway, Datadog, or custom webhook receiver.", - "wizard.monitoring.endpoint.placeholder": "https://prometheus.example.com/metrics OR env:DATADOG_URL", - "wizard.monitoring.metrics_export.label": "Metrics format", + "wizard.monitoring.title": "External monitoring plan (Art. 72 / 17(1)(h))", + "wizard.monitoring.hint": "Declare your external monitoring setup in the compliance manifest / Annex IV. Metadata only; ForgeLM runs no monitoring.", + "wizard.monitoring.enabled.label": "Record monitoring-plan metadata", + "wizard.monitoring.enabled.hint": "Optional for every risk tier. Recording a plan does not establish that monitoring is running.", + "wizard.monitoring.enabled.detail": "ForgeLM records the declared plan with declarative_only: true. It does not contact the endpoint, export metrics, detect drift, send drift alerts, or schedule checks. Operate your external stack separately; this declaration alone does not satisfy a post-market monitoring obligation.", + "wizard.monitoring.endpoint.label": "Declared endpoint (optional)", + "wizard.monitoring.endpoint.hint": "Recorded only, never contacted. env:NAME is stored as $NAME without reading the variable; no endpoint records null.", + "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com OR env:MONITORING_URL", + "wizard.monitoring.metrics_export.label": "Declared external exporter", "wizard.monitoring.metrics_export.none": "None", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "Custom webhook", - "wizard.monitoring.alert_on_drift.label": "Alert on drift", - "wizard.monitoring.alert_on_drift.hint": "Emit a webhook alert when the drift detector flags a regression.", + "wizard.monitoring.alert_on_drift.label": "Declare external drift-alert policy", + "wizard.monitoring.alert_on_drift.hint": "Records the policy of your external stack. ForgeLM does not detect drift or send these alerts.", "wizard.synthetic.title": "Synthetic data generation (advanced)", "wizard.synthetic.hint": "Generate teacher → student training data using a stronger model.", "wizard.synthetic.enabled.label": "Generate synthetic data", @@ -1834,7 +1834,6 @@ "wizard.error.model.format": "HuggingFace Hub IDs use the org/repo shape (e.g. meta-llama/Llama-3.1-8B-Instruct). A bare name without a slash won’t resolve at training time.", "wizard.error.dataset.required": "Pick a dataset before continuing — without a path or HuggingFace name the trainer has nothing to learn from.", "wizard.error.webhook.required": "You enabled notifications, so we need a webhook URL or env var to send them to. Either turn notifications off above or paste your Slack / Teams / Discord URL (or env:SLACK_WEBHOOK_URL for production secrets).", - "wizard.error.monitoring.endpoint.required": "You enabled post-market monitoring, so we need an endpoint to push metrics to. Either turn monitoring off above or paste your Prometheus / Datadog / custom-webhook URL (use env:VAR_NAME for secrets).", "wizard.error.synthetic.teacher_model.required": "You enabled synthetic data generation, so we need a teacher model name (e.g. gpt-4o, claude-opus-4-7, or a local model path).", "wizard.error.synthetic.seed_file.required": "You enabled synthetic data generation, so we need a seed prompts file — the teacher will produce a response for each prompt in it (e.g. ./data/seed_prompts.jsonl).", "wizard.error.synthetic.api_key_env.required": "You enabled synthetic data generation, so we need the env var that carries the teacher API key — typically OPENAI_API_KEY or ANTHROPIC_API_KEY. Leave the synthetic toggle off if you don’t have one.", @@ -2073,7 +2072,7 @@ "wizard.step.operations.title": "Operasyon ve gelişmiş ayarlar", "wizard.step.operations.name": "Operasyon", "wizard.step.operations.desc": "Bildirimler, post-market monitoring, sentetik veri üretimi ve saklama ufukları. Hepsi opsiyonel — ihtiyacın olmayanları atla.", - "wizard.step.operations.tutorial": "Bu dört blok, ForgeLM'in temel eğitim + compliance akışının ötesinde sağladığı her şeyi kapsar. Webhook, eğitim başladığında / başarılı olduğunda / hata verdiğinde Slack/Teams/Discord uyarısı gönderir. Post-market monitoring (Madde 12+17) deploy sonrasında drift detection'ı Prometheus / Datadog'a bağlar. Sentetik veri üretimi, daha güçlü bir teacher modelin ek eğitim örnekleri üretmesini sağlar. Saklama ufukları audit log'u ve ara artefact'ları için GDPR uyumlu TTL'ler ayarlar. Hiçbiri zorunlu değil — eğitimin ihtiyaç duymuyorsa kapalı bırak.", + "wizard.step.operations.tutorial": "Bu dört blok, ForgeLM'in temel eğitim + compliance akışının ötesinde sağladığı her şeyi kapsar. Webhook, eğitim başladığında / başarılı olduğunda / hata verdiğinde Slack/Teams/Discord uyarısı gönderir. İzleme (Madde 72 / 17(1)(h)) harici planı metadata olarak kaydeder; ForgeLM metrik aktarmaz, model sapması tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. Sentetik veri üretimi, daha güçlü bir teacher modelin ek eğitim örnekleri üretmesini sağlar. Saklama ufukları audit log'u ve ara artefact'ları için GDPR uyumlu TTL'ler ayarlar. Hiçbiri zorunlu değil — eğitimin ihtiyaç duymuyorsa kapalı bırak.", "wizard.risk.label": "EU AI Act risk sınıflandırması", "wizard.risk.hint": "Dağıtımın hangi seviyeye girdiğini seç. High-risk ve unacceptable, zorunlu denetim + insan onay yükümlülüklerini tetikler.", "wizard.risk.unknown": "Unknown — ön sınıflandırma yer tutucusu", @@ -2170,25 +2169,25 @@ "wizard.webhook.url.detail": "URL'yi doğrudan yapıştırabilirsin (gizli olmayan test kanalları için iyi) ya da bir ortam değişkenine işaret etmek için env: önekini kullan. Env-var yolu URL'yi YAML'den ve audit log'tan dışarıda tutar — production gizliler için zorunlu.", "wizard.webhook.enabled.label": "Webhook bildirimleri gönder", "wizard.webhook.enabled.hint": "Açıkken, ForgeLM seçtiğin olaylarda bir webhook payload gönderir. Kapalıyken YAML herhangi bir webhook bloğu içermez.", - "wizard.webhook.enabled.detail": "Bildirimler eğitim olaylarını ekip sohbetine (Slack / Teams / Discord) veya JSON kabul eden herhangi bir HTTPS endpoint'ine bağlar. Audit log her gönderimi — hatalar dahil — kaydeder, böylece neyin gönderildiğine dair bağımsız bir iz kalır. YAML'in webhook config taşımasını hiç istemiyorsan kapalı bırak.", + "wizard.webhook.enabled.detail": "Bildirimler eğitim olaylarını ekip sohbetine (Slack / Teams / Discord) veya JSON kabul eden herhangi bir HTTPS endpoint'ine bağlar. Teslimat hataları yalnızca süreç günlüğüne uyarı olarak yazılır. Audit log teslimat denemelerini değil, temel eğitim olaylarını kaydeder. YAML'in webhook config taşımasını hiç istemiyorsan kapalı bırak.", "wizard.webhook.notify_start.label": "Eğitim başında bildir", "wizard.webhook.notify_success.label": "Başarıda bildir", "wizard.webhook.notify_failure.label": "Hatada bildir", - "wizard.monitoring.title": "Post-market monitoring (Madde 12+17)", - "wizard.monitoring.hint": "Eğitim metriklerini sürekli observability için Prometheus / Datadog / özel webhook'a bağla.", - "wizard.monitoring.enabled.label": "Monitoring hook'ları aç", - "wizard.monitoring.enabled.hint": "High-risk sistemler için EU AI Act Madde 12 + 17 zorunlu kılar.", - "wizard.monitoring.enabled.detail": "High-risk bir sistem deploy edildikten sonra EU AI Act sürekli gözlem zorunlu kılar — drift detection, hata raporlama, periyodik doğruluk yeniden-kontrolleri. ForgeLM monitoring'i kendisi çalıştırmaz ama doğru hook'ları training run'a yerleştirir; mevcut observability stack'in alır.", - "wizard.monitoring.endpoint.label": "Monitoring endpoint", - "wizard.monitoring.endpoint.hint": "Prometheus push gateway, Datadog veya özel webhook receiver URL'si.", - "wizard.monitoring.endpoint.placeholder": "https://prometheus.example.com/metrics VEYA env:DATADOG_URL", - "wizard.monitoring.metrics_export.label": "Metrik formatı", + "wizard.monitoring.title": "Harici izleme planı (Madde 72 / 17(1)(h))", + "wizard.monitoring.hint": "Harici izleme planını uyumluluk manifestine / Annex IV’e kaydet. Yalnızca metadata; ForgeLM izleme çalıştırmaz.", + "wizard.monitoring.enabled.label": "İzleme planı metadatasını kaydet", + "wizard.monitoring.enabled.hint": "Tüm risk seviyelerinde isteğe bağlıdır. Planın kaydedilmesi izlemenin çalıştığını kanıtlamaz.", + "wizard.monitoring.enabled.detail": "ForgeLM beyan edilen planı declarative_only: true etiketiyle kaydeder. Endpoint’e bağlanmaz, metrik aktarmaz, model sapması tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. Harici altyapını ayrıca işlet; bu beyan pazar sonrası izleme yükümlülüğünü tek başına karşılamaz.", + "wizard.monitoring.endpoint.label": "Beyan edilen endpoint (isteğe bağlı)", + "wizard.monitoring.endpoint.hint": "Yalnızca kaydedilir, bağlantı kurulmaz. env:NAME, değişken okunmadan $NAME olarak saklanır; boş endpoint null yazılır.", + "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com VEYA env:MONITORING_URL", + "wizard.monitoring.metrics_export.label": "Beyan edilen harici aktarım sistemi", "wizard.monitoring.metrics_export.none": "Yok", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "Özel webhook", - "wizard.monitoring.alert_on_drift.label": "Drift'te uyar", - "wizard.monitoring.alert_on_drift.hint": "Drift detector regression bayrağı koyduğunda webhook uyarısı gönder.", + "wizard.monitoring.alert_on_drift.label": "Harici sapma uyarısı politikasını beyan et", + "wizard.monitoring.alert_on_drift.hint": "Harici altyapının politikasını kaydeder. ForgeLM model sapması tespit etmez veya bu uyarıları göndermez.", "wizard.synthetic.title": "Sentetik veri üretimi (gelişmiş)", "wizard.synthetic.hint": "Daha güçlü bir model kullanarak teacher → student eğitim verisi üret.", "wizard.synthetic.enabled.label": "Sentetik veri üret", @@ -2229,7 +2228,6 @@ "wizard.error.model.format": "HuggingFace Hub ID’leri org/repo formatında olmalı (örn. meta-llama/Llama-3.1-8B-Instruct). Slash içermeyen düz bir isim eğitim sırasında bulunamaz.", "wizard.error.dataset.required": "Devam etmeden önce bir dataset seç — yol veya HuggingFace ismi olmadan trainer’ın öğreneceği bir şey yok.", "wizard.error.webhook.required": "Bildirimleri açtın, yani göndereceğimiz bir webhook URL’si veya env değişkeni gerekli. Ya yukarıdan bildirimleri kapat ya da Slack / Teams / Discord URL’ini yapıştır (production gizliler için env:SLACK_WEBHOOK_URL kullan).", - "wizard.error.monitoring.endpoint.required": "Post-market monitoring’i açtın, yani metrikleri yollayacağımız bir endpoint gerekli. Ya yukarıdan monitoring’i kapat ya da Prometheus / Datadog / özel-webhook URL’ini yapıştır (gizliler için env:DEĞİŞKEN_ADI).", "wizard.error.synthetic.teacher_model.required": "Sentetik veri üretimini açtın, yani bir teacher model adı gerekli (örn. gpt-4o, claude-opus-4-7 veya yerel model yolu).", "wizard.error.synthetic.seed_file.required": "Sentetik veri üretimini açtın, yani bir seed prompts dosyası gerekli — teacher her prompt için yanıt üretir (örn. ./data/seed_prompts.jsonl).", "wizard.error.synthetic.api_key_env.required": "Sentetik veri üretimini açtın, yani teacher API anahtarını taşıyan env değişkeni gerekli — genellikle OPENAI_API_KEY veya ANTHROPIC_API_KEY. Bir tane yoksa sentetik anahtarını kapalı bırak.", @@ -2468,7 +2466,7 @@ "wizard.step.operations.title": "Betrieb & erweiterte Einstellungen", "wizard.step.operations.name": "Betrieb", "wizard.step.operations.desc": "Benachrichtigungen, Post-Market-Monitoring, synthetische Datengenerierung und Retentions-Horizonte. Alle optional — überspringen, was Sie nicht brauchen.", - "wizard.step.operations.tutorial": "Diese vier Blöcke decken alles ab, was ForgeLM jenseits des Kern-Training + Compliance-Flows liefert. Webhook sendet Slack/Teams/Discord-Alerts bei Trainingsstart/Erfolg/Fehler. Post-Market-Monitoring (Artikel 12+17) verdrahtet Drift-Detection nach Deployment in Prometheus/Datadog. Synthetische Datengenerierung lässt ein stärkeres Teacher-Modell zusätzliche Trainingsbeispiele produzieren. Retentions-Horizonte setzen DSGVO-konforme TTLs für Audit-Log und ephemere Artefakte. Nichts davon ist Pflicht — eingeklappt lassen wenn Ihr Run sie nicht braucht.", + "wizard.step.operations.tutorial": "Diese vier Blöcke decken alles ab, was ForgeLM jenseits des Kern-Training + Compliance-Flows liefert. Webhook sendet Slack/Teams/Discord-Alerts bei Trainingsstart/Erfolg/Fehler. Monitoring (Art. 72 / 17(1)(h)) erfasst einen externen Plan als Metadaten; ForgeLM exportiert keine Metriken, erkennt keine Drift, sendet keine Drift-Warnungen und plant keine Prüfungen. Synthetische Datengenerierung lässt ein stärkeres Teacher-Modell zusätzliche Trainingsbeispiele produzieren. Retentions-Horizonte setzen DSGVO-konforme TTLs für Audit-Log und ephemere Artefakte. Nichts davon ist Pflicht — eingeklappt lassen wenn Ihr Run sie nicht braucht.", "wizard.risk.label": "EU-AI-Act-Risikoklassifizierung", "wizard.risk.hint": "Wählen Sie die Klasse, unter die Ihr Deployment fällt. High-risk und unacceptable lösen verpflichtende Audit + Human-Approval-Pflichten aus.", "wizard.risk.unknown": "Unknown — Vorklassifizierungs-Platzhalter", @@ -2565,25 +2563,25 @@ "wizard.webhook.url.detail": "URL kann direkt eingefügt werden, oder via env:-Präfix auf eine Umgebungsvariable verweisen. Env-Variante hält die URL aus YAML + Audit-Log.", "wizard.webhook.enabled.label": "Webhook-Benachrichtigungen senden", "wizard.webhook.enabled.hint": "Wenn aktiv, sendet ForgeLM eine Webhook-Payload bei den ausgewählten Events. Aus = die YAML enthält keinen Webhook-Block.", - "wizard.webhook.enabled.detail": "Benachrichtigungen leiten Trainings-Events in Ihren Team-Chat (Slack / Teams / Discord) oder einen HTTPS-Endpoint, der JSON akzeptiert. Audit-Log-Einträge dokumentieren jeden Versand — auch Fehler — als unabhängige Spur. Aus lassen, wenn die YAML keinerlei Webhook-Config tragen soll.", + "wizard.webhook.enabled.detail": "Benachrichtigungen leiten Trainings-Events in Ihren Team-Chat (Slack / Teams / Discord) oder einen HTTPS-Endpoint, der JSON akzeptiert. Zustellfehler erscheinen nur als Warnungen im Prozesslog. Das Audit-Log erfasst die zugrunde liegenden Trainingsereignisse, keine Zustellversuche. Aus lassen, wenn die YAML keinerlei Webhook-Config tragen soll.", "wizard.webhook.notify_start.label": "Bei Trainingsstart benachrichtigen", "wizard.webhook.notify_success.label": "Bei Erfolg benachrichtigen", "wizard.webhook.notify_failure.label": "Bei Fehler benachrichtigen", - "wizard.monitoring.title": "Post-Market-Monitoring (Art. 12+17)", - "wizard.monitoring.hint": "Trainings-Metriken in Prometheus/Datadog/Custom-Webhook für laufende Observability.", - "wizard.monitoring.enabled.label": "Monitoring-Hooks aktivieren", - "wizard.monitoring.enabled.hint": "Pflicht für High-risk-Systeme nach EU AI Act Artikel 12 + 17.", - "wizard.monitoring.enabled.detail": "Nach Deployment eines High-risk-Systems verlangt der EU AI Act laufende Beobachtung — Drift-Detection, Fehler-Reporting, periodische Genauigkeitschecks.", - "wizard.monitoring.endpoint.label": "Monitoring-Endpoint", - "wizard.monitoring.endpoint.hint": "URL Ihres Prometheus-Push-Gateways, Datadogs oder Custom-Webhook-Receivers.", - "wizard.monitoring.endpoint.placeholder": "https://prometheus.example.com/metrics ODER env:DATADOG_URL", - "wizard.monitoring.metrics_export.label": "Metrik-Format", + "wizard.monitoring.title": "Externer Monitoring-Plan (Art. 72 / 17(1)(h))", + "wizard.monitoring.hint": "Externe Überwachung im Compliance-Manifest / Annex IV dokumentieren. Nur Metadaten; ForgeLM führt keine Überwachung aus.", + "wizard.monitoring.enabled.label": "Monitoring-Plan als Metadaten erfassen", + "wizard.monitoring.enabled.hint": "Für jede Risikostufe optional. Ein erfasster Plan belegt keine laufende Überwachung.", + "wizard.monitoring.enabled.detail": "ForgeLM erfasst den Plan mit declarative_only: true. Es kontaktiert keinen Endpoint, exportiert keine Metriken, erkennt keine Drift, sendet keine Drift-Warnungen und plant keine Prüfungen. Betreiben Sie Ihren externen Stack separat; die Erklärung allein erfüllt keine Pflicht zur Überwachung nach dem Inverkehrbringen.", + "wizard.monitoring.endpoint.label": "Deklarierter Endpoint (optional)", + "wizard.monitoring.endpoint.hint": "Nur erfasst, nie kontaktiert. env:NAME wird als $NAME gespeichert, ohne die Variable zu lesen; ohne Endpoint wird null erfasst.", + "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com ODER env:MONITORING_URL", + "wizard.monitoring.metrics_export.label": "Deklarierter externer Exporter", "wizard.monitoring.metrics_export.none": "Keine", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "Custom Webhook", - "wizard.monitoring.alert_on_drift.label": "Bei Drift alerten", - "wizard.monitoring.alert_on_drift.hint": "Webhook-Alert wenn Drift-Detector eine Regression markiert.", + "wizard.monitoring.alert_on_drift.label": "Externe Drift-Warnungen deklarieren", + "wizard.monitoring.alert_on_drift.hint": "Erfasst die Richtlinie Ihres externen Stacks. ForgeLM erkennt keine Drift und sendet diese Warnungen nicht.", "wizard.synthetic.title": "Synthetische Datengenerierung (erweitert)", "wizard.synthetic.hint": "Teacher → Student Trainingsdaten mit stärkerem Modell generieren.", "wizard.synthetic.enabled.label": "Synthetische Daten generieren", @@ -2624,7 +2622,6 @@ "wizard.error.model.format": "HuggingFace-Hub-IDs verwenden das Format org/repo (z. B. meta-llama/Llama-3.1-8B-Instruct). Ein Name ohne Schrägstrich wird zur Trainingszeit nicht aufgelöst.", "wizard.error.dataset.required": "Wählen Sie ein Dataset, bevor Sie fortfahren — ohne Pfad oder HuggingFace-Namen hat der Trainer nichts zum Lernen.", "wizard.error.webhook.required": "Sie haben Benachrichtigungen aktiviert — wir brauchen also eine Webhook-URL oder Env-Variable. Schalten Sie Benachrichtigungen oben aus oder fügen Sie Ihre Slack-/Teams-/Discord-URL ein (für Produktion env:SLACK_WEBHOOK_URL).", - "wizard.error.monitoring.endpoint.required": "Sie haben Post-Market-Monitoring aktiviert — wir brauchen einen Endpoint für Metriken. Schalten Sie Monitoring oben aus oder fügen Sie Ihre Prometheus-/Datadog-/Custom-Webhook-URL ein (für Secrets env:VAR_NAME).", "wizard.error.synthetic.teacher_model.required": "Sie haben synthetische Datengenerierung aktiviert — wir brauchen einen Teacher-Modellnamen (z. B. gpt-4o, claude-opus-4-7 oder lokaler Pfad).", "wizard.error.synthetic.seed_file.required": "Sie haben synthetische Datengenerierung aktiviert — wir brauchen eine Seed-Prompts-Datei (z. B. ./data/seed_prompts.jsonl); der Teacher generiert pro Prompt eine Antwort.", "wizard.error.synthetic.api_key_env.required": "Sie haben synthetische Datengenerierung aktiviert — wir brauchen die Env-Variable mit dem Teacher-API-Key (üblich: OPENAI_API_KEY oder ANTHROPIC_API_KEY).", @@ -2863,7 +2860,7 @@ "wizard.step.operations.title": "Opérations et paramètres avancés", "wizard.step.operations.name": "Opérations", "wizard.step.operations.desc": "Notifications, surveillance post-marché, génération de données synthétiques et horizons de rétention. Tous optionnels — passez ceux dont vous n'avez pas besoin.", - "wizard.step.operations.tutorial": "Ces quatre blocs couvrent tout ce que ForgeLM offre au-delà du flux core entraînement + compliance. Webhook envoie une alerte Slack/Teams/Discord au démarrage / succès / échec. La surveillance post-marché (Articles 12+17) connecte la détection de dérive à Prometheus / Datadog après déploiement. La génération synthétique laisse un modèle teacher plus fort produire des exemples supplémentaires. Les horizons de rétention fixent des TTL conformes RGPD pour audit log et artefacts éphémères. Aucun n'est obligatoire — laissez replié ce dont votre run n'a pas besoin.", + "wizard.step.operations.tutorial": "Ces quatre blocs couvrent tout ce que ForgeLM offre au-delà du flux core entraînement + compliance. Webhook envoie une alerte Slack/Teams/Discord au démarrage / succès / échec. La surveillance (Art. 72 / 17(1)(h)) enregistre un plan externe comme métadonnées ; ForgeLM n’exporte pas de métriques, ne détecte pas de dérive, n’envoie pas d’alertes de dérive et ne programme aucun contrôle. La génération synthétique laisse un modèle teacher plus fort produire des exemples supplémentaires. Les horizons de rétention fixent des TTL conformes RGPD pour audit log et artefacts éphémères. Aucun n'est obligatoire — laissez replié ce dont votre run n'a pas besoin.", "wizard.risk.label": "Classification de risque EU AI Act", "wizard.risk.hint": "Choisissez le niveau dans lequel votre déploiement tombe. High-risk et unacceptable déclenchent des obligations d'audit + approbation humaine obligatoires.", "wizard.risk.unknown": "Unknown — placeholder de pré-classification", @@ -2960,25 +2957,25 @@ "wizard.webhook.url.detail": "Vous pouvez coller URL directement, ou utiliser préfixe env: pour pointer une variable env.", "wizard.webhook.enabled.label": "Envoyer des notifications webhook", "wizard.webhook.enabled.hint": "Quand activé, ForgeLM publie un payload webhook aux événements sélectionnés. Désactivé = la YAML ne contient aucun bloc webhook.", - "wizard.webhook.enabled.detail": "Les notifications connectent les événements d'entraînement à votre chat d'équipe (Slack / Teams / Discord) ou tout endpoint HTTPS acceptant du JSON. Les entrées de l'audit log enregistrent chaque envoi — y compris les échecs — pour une piste indépendante. Laissez désactivé si vous ne voulez aucune config webhook dans la YAML.", + "wizard.webhook.enabled.detail": "Les notifications connectent les événements d'entraînement à votre chat d'équipe (Slack / Teams / Discord) ou tout endpoint HTTPS acceptant du JSON. Les échecs de livraison sont uniquement des avertissements dans le journal du processus. L’audit log enregistre les événements d’entraînement sous-jacents, pas les tentatives de livraison. Laissez désactivé si vous ne voulez aucune config webhook dans la YAML.", "wizard.webhook.notify_start.label": "Notifier au démarrage", "wizard.webhook.notify_success.label": "Notifier au succès", "wizard.webhook.notify_failure.label": "Notifier à l'échec", - "wizard.monitoring.title": "Surveillance post-marché (Art. 12+17)", - "wizard.monitoring.hint": "Branchez métriques d'entraînement à Prometheus/Datadog/webhook custom.", - "wizard.monitoring.enabled.label": "Activer hooks monitoring", - "wizard.monitoring.enabled.hint": "Obligatoire pour systèmes high-risk selon EU AI Act Articles 12 + 17.", - "wizard.monitoring.enabled.detail": "Une fois système high-risk déployé, EU AI Act exige observation continue — détection de drift, reporting d'erreurs.", - "wizard.monitoring.endpoint.label": "Endpoint monitoring", - "wizard.monitoring.endpoint.hint": "URL de votre Prometheus push gateway, Datadog ou récepteur webhook custom.", - "wizard.monitoring.endpoint.placeholder": "https://prometheus.example.com/metrics OU env:DATADOG_URL", - "wizard.monitoring.metrics_export.label": "Format métriques", + "wizard.monitoring.title": "Plan de surveillance externe (Art. 72 / 17(1)(h))", + "wizard.monitoring.hint": "Déclarez votre surveillance externe dans le manifeste de conformité / Annexe IV. Métadonnées uniquement ; ForgeLM ne surveille pas le modèle.", + "wizard.monitoring.enabled.label": "Enregistrer les métadonnées du plan", + "wizard.monitoring.enabled.hint": "Facultatif pour chaque niveau de risque. Un plan enregistré ne prouve pas que la surveillance fonctionne.", + "wizard.monitoring.enabled.detail": "ForgeLM enregistre le plan avec declarative_only: true. Il ne contacte pas le point de terminaison, n’exporte pas de métriques, ne détecte pas de dérive, n’envoie pas d’alertes de dérive et ne programme aucun contrôle. Exploitez votre infrastructure externe séparément ; cette déclaration seule ne remplit pas une obligation de surveillance après commercialisation.", + "wizard.monitoring.endpoint.label": "Point de terminaison déclaré (facultatif)", + "wizard.monitoring.endpoint.hint": "Enregistré uniquement, jamais contacté. env:NAME devient $NAME sans lire la variable ; sans adresse, la valeur est null.", + "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com OU env:MONITORING_URL", + "wizard.monitoring.metrics_export.label": "Exportateur externe déclaré", "wizard.monitoring.metrics_export.none": "Aucun", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "Webhook custom", - "wizard.monitoring.alert_on_drift.label": "Alerte sur drift", - "wizard.monitoring.alert_on_drift.hint": "Émettre alert webhook quand détecteur drift signale régression.", + "wizard.monitoring.alert_on_drift.label": "Déclarer les alertes de dérive externes", + "wizard.monitoring.alert_on_drift.hint": "Enregistre la politique de votre infrastructure externe. ForgeLM ne détecte pas de dérive et n’envoie pas ces alertes.", "wizard.synthetic.title": "Génération données synthétiques (avancé)", "wizard.synthetic.hint": "Générez données entraînement teacher → student avec modèle plus fort.", "wizard.synthetic.enabled.label": "Générer données synthétiques", @@ -3019,7 +3016,6 @@ "wizard.error.model.format": "Les ID HuggingFace Hub utilisent le format org/repo (ex. meta-llama/Llama-3.1-8B-Instruct). Un nom sans barre oblique ne sera pas résolu lors de l’entraînement.", "wizard.error.dataset.required": "Choisissez un dataset avant de continuer — sans chemin ni nom HuggingFace, le trainer n’a rien à apprendre.", "wizard.error.webhook.required": "Vous avez activé les notifications — il nous faut donc une URL webhook ou une variable d’env. Désactivez les notifications ci-dessus ou collez votre URL Slack / Teams / Discord (pour la production utilisez env:SLACK_WEBHOOK_URL).", - "wizard.error.monitoring.endpoint.required": "Vous avez activé la surveillance post-marché — il nous faut un endpoint pour les métriques. Désactivez la surveillance ci-dessus ou collez votre URL Prometheus / Datadog / webhook custom (pour les secrets utilisez env:VAR_NAME).", "wizard.error.synthetic.teacher_model.required": "Vous avez activé la génération de données synthétiques — il nous faut un nom de modèle teacher (ex. gpt-4o, claude-opus-4-7 ou chemin local).", "wizard.error.synthetic.seed_file.required": "Vous avez activé la génération de données synthétiques — il nous faut un fichier de seed prompts (ex. ./data/seed_prompts.jsonl) ; le teacher produira une réponse par prompt.", "wizard.error.synthetic.api_key_env.required": "Vous avez activé la génération de données synthétiques — il nous faut la variable d’env qui porte la clé API du teacher (généralement OPENAI_API_KEY ou ANTHROPIC_API_KEY).", @@ -3258,7 +3254,7 @@ "wizard.step.operations.title": "Operaciones y ajustes avanzados", "wizard.step.operations.name": "Operaciones", "wizard.step.operations.desc": "Notificaciones, monitoreo post-mercado, generación de datos sintéticos y horizontes de retención. Todos opcionales — salte los que no necesite.", - "wizard.step.operations.tutorial": "Estos cuatro bloques cubren todo lo que ForgeLM ofrece más allá del flujo central de entrenamiento + compliance. Webhook envía alerta Slack/Teams/Discord en inicio / éxito / fallo. El monitoreo post-mercado (Artículos 12+17) conecta detección de drift a Prometheus / Datadog tras despliegue. La generación sintética permite a un modelo teacher más fuerte producir ejemplos adicionales. Los horizontes de retención fijan TTLs conformes a RGPD para audit log y artefactos efímeros. Ninguno es obligatorio — déjelos cerrados si su run no los necesita.", + "wizard.step.operations.tutorial": "Estos cuatro bloques cubren todo lo que ForgeLM ofrece más allá del flujo central de entrenamiento + compliance. Webhook envía alerta Slack/Teams/Discord en inicio / éxito / fallo. El monitoreo (Art. 72 / 17(1)(h)) registra un plan externo como metadatos; ForgeLM no exporta métricas, detecta deriva, envía alertas de deriva ni programa comprobaciones. La generación sintética permite a un modelo teacher más fuerte producir ejemplos adicionales. Los horizontes de retención fijan TTLs conformes a RGPD para audit log y artefactos efímeros. Ninguno es obligatorio — déjelos cerrados si su run no los necesita.", "wizard.risk.label": "Clasificación de riesgo EU AI Act", "wizard.risk.hint": "Elija el nivel en el que cae su despliegue. High-risk y unacceptable activan obligaciones obligatorias de auditoría + aprobación humana.", "wizard.risk.unknown": "Unknown — placeholder de pre-clasificación", @@ -3355,25 +3351,25 @@ "wizard.webhook.url.detail": "Puede pegar URL directamente, o usar prefijo env: para apuntar a una variable de entorno.", "wizard.webhook.enabled.label": "Enviar notificaciones webhook", "wizard.webhook.enabled.hint": "Cuando está activo, ForgeLM publica un payload webhook en los eventos seleccionados. Apagado = la YAML no contiene ningún bloque webhook.", - "wizard.webhook.enabled.detail": "Las notificaciones conectan eventos de entrenamiento a su chat de equipo (Slack / Teams / Discord) o cualquier endpoint HTTPS que acepte JSON. Las entradas del audit log registran cada envío — incluyendo fallos — como un rastro independiente. Déjelo apagado si no quiere ninguna config de webhook en la YAML.", + "wizard.webhook.enabled.detail": "Las notificaciones conectan eventos de entrenamiento a su chat de equipo (Slack / Teams / Discord) o cualquier endpoint HTTPS que acepte JSON. Los fallos de entrega solo se registran como advertencias en el registro del proceso. El audit log registra los eventos de entrenamiento subyacentes, no los intentos de entrega. Déjelo apagado si no quiere ninguna config de webhook en la YAML.", "wizard.webhook.notify_start.label": "Notificar al inicio", "wizard.webhook.notify_success.label": "Notificar en éxito", "wizard.webhook.notify_failure.label": "Notificar en fallo", - "wizard.monitoring.title": "Monitoreo post-mercado (Art. 12+17)", - "wizard.monitoring.hint": "Conecte métricas entrenamiento a Prometheus/Datadog/webhook custom para observabilidad continua.", - "wizard.monitoring.enabled.label": "Habilitar hooks de monitoreo", - "wizard.monitoring.enabled.hint": "Obligatorio para sistemas high-risk bajo EU AI Act Artículos 12 + 17.", - "wizard.monitoring.enabled.detail": "Una vez desplegado un sistema high-risk, EU AI Act requiere observación continua — detección de drift, reporte de errores.", - "wizard.monitoring.endpoint.label": "Endpoint monitoreo", - "wizard.monitoring.endpoint.hint": "URL de su Prometheus push gateway, Datadog o receptor webhook custom.", - "wizard.monitoring.endpoint.placeholder": "https://prometheus.example.com/metrics O env:DATADOG_URL", - "wizard.monitoring.metrics_export.label": "Formato métricas", + "wizard.monitoring.title": "Plan de monitoreo externo (Art. 72 / 17(1)(h))", + "wizard.monitoring.hint": "Declare su monitoreo externo en el manifiesto de cumplimiento / Anexo IV. Solo metadatos; ForgeLM no ejecuta el monitoreo.", + "wizard.monitoring.enabled.label": "Registrar metadatos del plan", + "wizard.monitoring.enabled.hint": "Opcional para todos los niveles de riesgo. Registrar un plan no demuestra que el monitoreo esté funcionando.", + "wizard.monitoring.enabled.detail": "ForgeLM registra el plan con declarative_only: true. No contacta el endpoint, exporta métricas, detecta deriva, envía alertas de deriva ni programa comprobaciones. Opere su infraestructura externa por separado; esta declaración por sí sola no cumple una obligación de monitoreo posterior a la comercialización.", + "wizard.monitoring.endpoint.label": "Endpoint declarado (opcional)", + "wizard.monitoring.endpoint.hint": "Solo se registra, nunca se contacta. env:NAME se guarda como $NAME sin leer la variable; sin endpoint se registra null.", + "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com O env:MONITORING_URL", + "wizard.monitoring.metrics_export.label": "Exportador externo declarado", "wizard.monitoring.metrics_export.none": "Ninguno", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "Webhook custom", - "wizard.monitoring.alert_on_drift.label": "Alertar en drift", - "wizard.monitoring.alert_on_drift.hint": "Emite alerta webhook cuando detector drift marca regresión.", + "wizard.monitoring.alert_on_drift.label": "Declarar alertas externas de deriva", + "wizard.monitoring.alert_on_drift.hint": "Registra la política de su infraestructura externa. ForgeLM no detecta deriva ni envía estas alertas.", "wizard.synthetic.title": "Generación datos sintéticos (avanzado)", "wizard.synthetic.hint": "Genere datos entrenamiento teacher → student usando modelo más fuerte.", "wizard.synthetic.enabled.label": "Generar datos sintéticos", @@ -3414,7 +3410,6 @@ "wizard.error.model.format": "Los ID de HuggingFace Hub usan el formato org/repo (ej. meta-llama/Llama-3.1-8B-Instruct). Un nombre sin barra no se resolverá al entrenar.", "wizard.error.dataset.required": "Elija un dataset antes de continuar — sin una ruta o nombre de HuggingFace, el trainer no tiene de qué aprender.", "wizard.error.webhook.required": "Activó las notificaciones, así que necesitamos una URL de webhook o variable de entorno. Desactive las notificaciones arriba o pegue su URL de Slack / Teams / Discord (para producción use env:SLACK_WEBHOOK_URL).", - "wizard.error.monitoring.endpoint.required": "Activó el monitoreo post-mercado, así que necesitamos un endpoint para enviar métricas. Desactive el monitoreo arriba o pegue su URL de Prometheus / Datadog / webhook custom (para secretos use env:NOMBRE_VAR).", "wizard.error.synthetic.teacher_model.required": "Activó la generación de datos sintéticos, así que necesitamos un nombre de modelo teacher (ej. gpt-4o, claude-opus-4-7 o ruta local).", "wizard.error.synthetic.seed_file.required": "Activó la generación de datos sintéticos, así que necesitamos un archivo de seed prompts (ej. ./data/seed_prompts.jsonl); el teacher producirá una respuesta por cada prompt.", "wizard.error.synthetic.api_key_env.required": "Activó la generación de datos sintéticos, así que necesitamos la variable de entorno con la clave API del teacher (típicamente OPENAI_API_KEY o ANTHROPIC_API_KEY).", @@ -3653,7 +3648,7 @@ "wizard.step.operations.title": "运维与高级设置", "wizard.step.operations.name": "运维", "wizard.step.operations.desc": "通知、上市后监控、合成数据生成和保留期限。全部可选 — 不需要的可以跳过。", - "wizard.step.operations.tutorial": "这四个区块涵盖了 ForgeLM 在核心训练 + 合规流程之外提供的一切。Webhook 在训练开始 / 成功 / 失败时发送 Slack/Teams/Discord 警报。上市后监控(第 12+17 条)在部署后将漂移检测连接到 Prometheus / Datadog。合成数据生成让更强的 teacher 模型产生额外训练样本。保留期限为审计日志和临时 artefact 设置符合 GDPR 的 TTL。没有任何项是强制的 — 你的训练不需要的话保持折叠状态即可。", + "wizard.step.operations.tutorial": "这四个区块涵盖了 ForgeLM 在核心训练 + 合规流程之外提供的一切。Webhook 在训练开始 / 成功 / 失败时发送 Slack/Teams/Discord 警报。监控（第 72 / 17(1)(h) 条）仅将外部计划记录为元数据；ForgeLM 不导出指标、检测漂移、发送漂移警报或安排检查。合成数据生成让更强的 teacher 模型产生额外训练样本。保留期限为审计日志和临时 artefact 设置符合 GDPR 的 TTL。没有任何项是强制的 — 你的训练不需要的话保持折叠状态即可。", "wizard.risk.label": "EU AI Act 风险分类", "wizard.risk.hint": "选择你的部署所属的等级。High-risk 和 unacceptable 触发强制审计 + 人工批准义务。", "wizard.risk.unknown": "Unknown — 预分类占位符", @@ -3750,25 +3745,25 @@ "wizard.webhook.url.detail": "你可以直接粘贴 URL,或使用 env: 前缀指向环境变量。", "wizard.webhook.enabled.label": "发送 webhook 通知", "wizard.webhook.enabled.hint": "启用后,ForgeLM 在你选择的事件发送 webhook payload。关闭 = YAML 不包含任何 webhook 块。", - "wizard.webhook.enabled.detail": "通知将训练事件连接到你的团队聊天(Slack / Teams / Discord)或任何接受 JSON 的 HTTPS endpoint。审计日志条目记录每次发送 — 包括失败 — 提供独立追踪。如果不想 YAML 中包含任何 webhook 配置,保持关闭。", + "wizard.webhook.enabled.detail": "通知将训练事件连接到你的团队聊天(Slack / Teams / Discord)或任何接受 JSON 的 HTTPS endpoint。发送失败仅在进程日志中记录为警告。审计日志记录对应的训练事件，不记录发送尝试。如果不想 YAML 中包含任何 webhook 配置,保持关闭。", "wizard.webhook.notify_start.label": "训练开始时通知", "wizard.webhook.notify_success.label": "成功时通知", "wizard.webhook.notify_failure.label": "失败时通知", - "wizard.monitoring.title": "上市后监控(第 12+17 条)", - "wizard.monitoring.hint": "将训练指标连接到 Prometheus/Datadog/自定义 webhook 以持续观测。", - "wizard.monitoring.enabled.label": "启用监控 hook", - "wizard.monitoring.enabled.hint": "EU AI Act 第 12 + 17 条要求 high-risk 系统强制启用。", - "wizard.monitoring.enabled.detail": "high-risk 系统部署后,EU AI Act 要求持续观察 — 漂移检测、错误报告、定期准确性复查。", - "wizard.monitoring.endpoint.label": "监控 endpoint", - "wizard.monitoring.endpoint.hint": "你的 Prometheus push gateway、Datadog 或自定义 webhook 接收器的 URL。", - "wizard.monitoring.endpoint.placeholder": "https://prometheus.example.com/metrics 或 env:DATADOG_URL", - "wizard.monitoring.metrics_export.label": "指标格式", + "wizard.monitoring.title": "外部监控计划（第 72 / 17(1)(h) 条）", + "wizard.monitoring.hint": "在合规清单 / 附录 IV 中声明外部监控设置。仅记录元数据；ForgeLM 不运行监控。", + "wizard.monitoring.enabled.label": "记录监控计划元数据", + "wizard.monitoring.enabled.hint": "所有风险级别均可选。记录计划并不能证明监控正在运行。", + "wizard.monitoring.enabled.detail": "ForgeLM 使用 declarative_only: true 标记记录计划。它不会连接端点、导出指标、检测漂移、发送漂移警报或安排检查。请单独运行外部监控系统；仅此声明并不能履行上市后监控义务。", + "wizard.monitoring.endpoint.label": "声明的端点（可选）", + "wizard.monitoring.endpoint.hint": "仅记录，不连接。env:NAME 保存为 $NAME，不读取变量；未提供端点时记录 null。", + "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com 或 env:MONITORING_URL", + "wizard.monitoring.metrics_export.label": "声明的外部指标导出器", "wizard.monitoring.metrics_export.none": "无", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "自定义 webhook", - "wizard.monitoring.alert_on_drift.label": "漂移时报警", - "wizard.monitoring.alert_on_drift.hint": "当漂移检测器标记回归时发出 webhook 警报。", + "wizard.monitoring.alert_on_drift.label": "声明外部漂移警报策略", + "wizard.monitoring.alert_on_drift.hint": "记录外部系统的策略。ForgeLM 不检测漂移，也不发送这些警报。", "wizard.synthetic.title": "合成数据生成(高级)", "wizard.synthetic.hint": "使用更强的模型生成 teacher → student 训练数据。", "wizard.synthetic.enabled.label": "生成合成数据", @@ -3809,7 +3804,6 @@ "wizard.error.model.format": "HuggingFace Hub ID 使用 org/repo 格式(如 meta-llama/Llama-3.1-8B-Instruct)。不带斜杠的纯名称在训练时无法解析。", "wizard.error.dataset.required": "继续之前请选择一个数据集 — 没有路径或 HuggingFace 名称,trainer 没有可学习的对象。", "wizard.error.webhook.required": "你启用了通知,所以需要一个 webhook URL 或环境变量。要么关闭上方的通知,要么粘贴你的 Slack / Teams / Discord URL(生产环境密钥请使用 env:SLACK_WEBHOOK_URL)。", - "wizard.error.monitoring.endpoint.required": "你启用了上市后监控,所以需要一个推送指标的 endpoint。要么关闭上方的监控,要么粘贴你的 Prometheus / Datadog / 自定义 webhook URL(密钥请使用 env:变量名)。", "wizard.error.synthetic.teacher_model.required": "你启用了合成数据生成,所以需要一个 teacher 模型名(如 gpt-4o、claude-opus-4-7 或本地模型路径)。", "wizard.error.synthetic.seed_file.required": "你启用了合成数据生成,所以需要一个 seed prompts 文件(如 ./data/seed_prompts.jsonl);teacher 会为每个 prompt 生成一个回应。", "wizard.error.synthetic.api_key_env.required": "你启用了合成数据生成,所以需要承载 teacher API 密钥的环境变量(通常是 OPENAI_API_KEY 或 ANTHROPIC_API_KEY)。", diff --git a/site/js/wizard.js b/site/js/wizard.js index 87ffb806..71c05d37 100644 --- a/site/js/wizard.js +++ b/site/js/wizard.js @@ -251,8 +251,7 @@ webhookNotifyStart: false, webhookNotifySuccess: true, webhookNotifyFailure: true, - // Article 12+17 post-market monitoring. Auto-coerced to true - // for high-risk tiers. + // External monitoring-plan metadata; opt-in for every risk tier. monitoringEnabled: false, monitoringEndpoint: '', metricsExport: 'none', // 'none' | 'prometheus' | 'datadog' | 'custom_webhook' @@ -448,9 +447,6 @@ if (state.webhookEnabled && blank(state.webhookUrl)) { errors.push({ field: 'webhookUrl', key: 'wizard.error.webhook.required' }); } - if (state.monitoringEnabled && blank(state.monitoringEndpoint)) { - errors.push({ field: 'monitoringEndpoint', key: 'wizard.error.monitoring.endpoint.required' }); - } if (state.syntheticEnabled) { if (blank(state.syntheticTeacherModel)) { errors.push({ field: 'syntheticTeacherModel', key: 'wizard.error.synthetic.teacher_model.required' }); @@ -818,10 +814,10 @@ blank(); } - // Article 12+17 post-market monitoring. Auto-coerced enabled for - // strict tiers (operator can't disable it under high-risk). - var monitoringActive = state.monitoringEnabled || isStrictRisk(state); + // Record an external plan without activating monitoring hooks. + var monitoringActive = state.monitoringEnabled; if (monitoringActive || state.monitoringEndpoint) { + comment('External monitoring plan metadata only; ForgeLM runs no exporters, drift alerts, or scheduled checks.'); header(0, 'monitoring'); k(1, 'enabled', monitoringActive ? 'true' : 'false'); if (state.monitoringEndpoint) { @@ -2701,26 +2697,17 @@ body.appendChild(makeWebhookToggle('webhookNotifyFailure', 'wizard.webhook.notify_failure.label')); })); - // Post-market monitoring (Article 12+17) accordion. + // Declarative external monitoring plan (Article 72 / 17(1)(h)). pane.appendChild(makeAccordion(state, persist, 'monitoring', 'wizard.monitoring.title', 'wizard.monitoring.hint', function (body) { - var monLocked = isStrictRisk(state); var monInput = el('input', { type: 'checkbox' }); - monInput.checked = !!state.monitoringEnabled || monLocked; - if (monLocked) monInput.disabled = true; + monInput.checked = !!state.monitoringEnabled; monInput.addEventListener('change', function () { state.monitoringEnabled = monInput.checked; persist(); rerender(); }); var monLabelEl = el('span', { class: 'wizard-toggle-label', text: tr('wizard.monitoring.enabled.label') }); - if (monLocked) { - monLabelEl.appendChild(el('span', { - class: 'badge', - text: tr('wizard.locked.badge'), - style: 'margin-left: 0.4rem;' - })); - } var monRow = el('div', { class: 'wizard-row' }, [ el('label', { class: 'wizard-toggle' }, [monInput, monLabelEl]), el('span', { class: 'wizard-row-hint', text: tr('wizard.monitoring.enabled.hint') }) @@ -2728,15 +2715,11 @@ appendDetail(monRow, 'wizard.monitoring.enabled.detail'); body.appendChild(monRow); - var endpointShown = !!state.monitoringEnabled || monLocked; + var endpointShown = !!state.monitoringEnabled; if (endpointShown) { - // Endpoint is required when monitoring is enabled — same - // pattern as webhook.url. fieldHasError gates the visual so - // the error appears only after a failed Next attempt. - var monEndpointErrKey = fieldHasError(state, 'operations', 'monitoringEndpoint'); var epInput = el('input', { type: 'text', - class: 'wizard-input' + (monEndpointErrKey ? ' is-invalid' : ''), + class: 'wizard-input', placeholder: tr('wizard.monitoring.endpoint.placeholder'), value: state.monitoringEndpoint || '' }); @@ -2745,20 +2728,11 @@ epInput, el('span', { class: 'wizard-row-hint', text: tr('wizard.monitoring.endpoint.hint') }) ]; - if (monEndpointErrKey) { - epRowChildren.push(el('div', { class: 'wizard-error', role: 'alert', text: tr(monEndpointErrKey) })); - } var epRow = el('div', { class: 'wizard-row' }, epRowChildren); epInput.addEventListener('input', function () { state.monitoringEndpoint = epInput.value.trim(); persist(); liveYaml(); - var stillErr = validateStep(state, 'operations').some(function (e) { return e.field === 'monitoringEndpoint'; }); - if (!stillErr) { - epInput.classList.remove('is-invalid'); - var errEl = epRow.querySelector('.wizard-error'); - if (errEl) errEl.remove(); - } }); body.appendChild(epRow); diff --git a/tests/test_config.py b/tests/test_config.py index ae2a0366..903390f5 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -79,6 +79,31 @@ def test_unsloth_backend(self): assert m.backend == "unsloth" +class TestMonitoringConfig: + @pytest.mark.parametrize("exporter", ["prometheus", "datadog", "custom_webhook"]) + def test_exporter_declaration_warns_at_config_load(self, tmp_path, caplog, exporter): + config_data = minimal_config( + monitoring={"enabled": True, "metrics_export": exporter, "endpoint": "https://secret.example.com/private"} + ) + path = _write_yaml(config_data, str(tmp_path / "monitoring.yaml")) + with caplog.at_level(logging.WARNING, logger="forgelm.config"): + config = load_config(path) + assert config.monitoring.metrics_export == exporter + assert "monitoring.metrics_export" in caplog.text + assert "declarative metadata only" in caplog.text + assert "does not export metrics" in caplog.text + assert "secret.example.com" not in caplog.text + + @pytest.mark.parametrize( + "monitoring", + [None, {"enabled": False, "metrics_export": "prometheus"}, {"enabled": True, "metrics_export": "none"}], + ) + def test_no_exporter_warning_for_inactive_export(self, caplog, monitoring): + with caplog.at_level(logging.WARNING, logger="forgelm.config"): + ForgeConfig(**minimal_config(monitoring=monitoring)) + assert "monitoring.metrics_export" not in caplog.text + + # --- LoraConfigModel --- diff --git a/tests/test_eu_ai_act.py b/tests/test_eu_ai_act.py index 58ef5e30..9a102a06 100644 --- a/tests/test_eu_ai_act.py +++ b/tests/test_eu_ai_act.py @@ -11,6 +11,7 @@ from forgelm.compliance import ( AuditLogger, + compute_annex_iv_manifest_hash, export_compliance_artifacts, export_evidence_bundle, generate_deployer_instructions, @@ -815,6 +816,60 @@ def test_successful_export_promotes_all_artifacts(self, tmp_path): class TestManifestAnnexIV: + @pytest.mark.parametrize("env_value", [None, "https://secret.example.com/private"]) + @pytest.mark.parametrize( + "endpoint_fields,expected", + [ + ({}, None), + ({"endpoint": "https://monitor.example.com"}, "https://monitor.example.com"), + ({"endpoint_env": "MON_URL"}, "$MON_URL"), + ({"endpoint": "https://monitor.example.com", "endpoint_env": "MON_URL"}, "$MON_URL"), + ({"endpoint_env": ""}, None), + ], + ) + def test_monitoring_declaration_survives_export( + self, tmp_path, minimal_config, monkeypatch, endpoint_fields, expected, env_value + ): + """A plan is labelled as metadata and never resolves endpoint secrets.""" + if env_value is None: + monkeypatch.delenv("MON_URL", raising=False) + else: + monkeypatch.setenv("MON_URL", env_value) + config = ForgeConfig( + **minimal_config( + compliance={"provider_name": "Corp", "system_name": "Bot", "intended_purpose": "Testing"}, + monitoring={ + "enabled": True, + "metrics_export": "prometheus", + "alert_on_drift": True, + "check_interval_hours": 12, + **endpoint_fields, + }, + ) + ) + manifest = generate_training_manifest(config, {"eval_loss": 0.5}) + declaration = { + "declarative_only": True, + "endpoint": expected, + "metrics_export": "prometheus", + "alert_on_drift": True, + "check_interval_hours": 12, + } + assert manifest["monitoring"] == declaration + export_compliance_artifacts(manifest, str(tmp_path)) + report = json.loads((tmp_path / "compliance_report.json").read_text(encoding="utf-8")) + annex = json.loads((tmp_path / "annex_iv_metadata.json").read_text(encoding="utf-8")) + assert report["monitoring"] == declaration + assert annex["monitoring_and_logging"] == declaration + assert "secret.example.com" not in json.dumps(report) + assert "secret.example.com" not in json.dumps(annex) + assert annex["metadata"]["manifest_hash"] == compute_annex_iv_manifest_hash(annex) + + @pytest.mark.parametrize("monitoring", [None, {"enabled": False, "metrics_export": "datadog"}]) + def test_disabled_monitoring_is_not_declared(self, minimal_config, monitoring): + config = ForgeConfig(**minimal_config(monitoring=monitoring)) + assert "monitoring" not in generate_training_manifest(config, {}) + def test_includes_annex_iv(self, minimal_config): config = ForgeConfig( **minimal_config( diff --git a/tests/test_wizard_phase22.py b/tests/test_wizard_phase22.py index a1188788..107afc24 100644 --- a/tests/test_wizard_phase22.py +++ b/tests/test_wizard_phase22.py @@ -1190,6 +1190,15 @@ def test_minimal_config_prints_checklist(self, capsys, monkeypatch): class TestMonitoringEnvPrefix: """P9 — monitoring collector accepts env:VAR_NAME like the webhook collector.""" + def test_metadata_plan_accepts_no_endpoint(self, capsys): + with patch("builtins.input", side_effect=_input_returning("y", "", "1", "y", "24")): + result = wizard._collect_monitoring() + assert result == {"enabled": True, "metrics_export": "none", "alert_on_drift": True, "check_interval_hours": 24} + output = capsys.readouterr().out + assert "Metadata only" in output + assert "does not export metrics" in output + assert "schedule checks" in output + def test_env_prefix_routes_to_endpoint_env(self): with patch( "builtins.input", diff --git a/tools/check_module_size.py b/tools/check_module_size.py index 64027298..4feb73da 100644 --- a/tools/check_module_size.py +++ b/tools/check_module_size.py @@ -263,7 +263,7 @@ class _DeferredSplit: # _validate_entries() sees a raise without reading the diff. _DEFERRED_SPLITS: dict[str, _DeferredSplit] = { "forgelm/compliance.py": _DeferredSplit( - budget=2492, + budget=2493, deferred_at_loc=2147, reason=( "EU AI Act Art. 9-17 + Annex IV builder + hash-chained audit log + " @@ -271,6 +271,10 @@ class _DeferredSplit: "_annex_iv, _provenance, _gdpr." ), budget_history=( + "2026-10-15 (#93, 2492 -> 2493, +1): label the persisted monitoring plan " + "declarative_only so neither the manifest nor its Annex IV projection implies " + "runtime monitoring. Keep the label with the existing writer; the one-line " + "evidence correction does not warrant an unrelated compliance-package split.", "2026-10-09 (Phase 16 S1-S3 review, 2473 -> 2492, +19): " "`match_annex_iv_manifest_hash`. Hashing the Annex IV / pipeline manifest over " "the strict-JSON form (so a fresh artefact holding a NaN verifies) meant an " @@ -313,9 +317,14 @@ class _DeferredSplit: ), ), "forgelm/config.py": _DeferredSplit( - budget=1855, + budget=1870, deferred_at_loc=1795, budget_history=( + "2026-10-15 (#93, 1855 -> 1870, +15): describe MonitoringConfig as external-plan " + "metadata and warn when an enabled plan declares an exporter ForgeLM does not run. " + "The validator belongs beside its schema; the remaining growth is required field " + "description wrapping. Keep this contract repair separate from the deferred " + "schema split, whose import-time validation order needs its own review.", "2026-10-09 (Phase 16 S1-S3 review, 1846 -> 1855, +9): the `FiniteFloat` alias " "(`Annotated[float, allow_inf_nan=False]`, 7 lines with its rationale) and its " "application to the twelve float fields S2 left open — `learning_rate`, " From 2f602b635cb8ae5f0f9bcc0178639bf6bef00437 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 15 Oct 2026 22:04:55 +0300 Subject: [PATCH 02/44] fix: preserve monitoring declaration provenance and input contracts Add defaulted_fields and endpoint_redacted to the evidence projection, mask literal URLs with the shared scheme/host policy, and validate unresolved ASCII env names. Emit exporter warnings at file load rather than each pipeline validation. Keep schema defaults and existing input types. Align CLI/browser env parsing, omit disabled browser plans, expose the external check interval, and add Node execution/schema regressions to CI and the canonical gauntlets. Expand export-to-verifier and warning-count coverage, plus EN/TR input, migration, and evidence examples. Resolves review gaps in CR-A01-007, DOC-BFIELD-031, DOC-D3B04A-014, TECH-T09-003, DOC-BFIELD-017, DOC-BLEGAL-048, DOC-D4B09-009. Refs #93, #946 Co-authored-by: Codex --- .github/workflows/ci.yml | 8 + AGENTS.md | 1 + CHANGELOG.md | 25 ++- CLAUDE.md | 1 + CONTRIBUTING.md | 1 + config_template.yaml | 5 +- docs/reference/configuration-tr.md | 44 +++++- docs/reference/configuration.md | 37 ++++- docs/standards/testing.md | 8 + .../usermanuals/en/reference/configuration.md | 32 +++- .../usermanuals/tr/reference/configuration.md | 46 ++++-- forgelm/compliance.py | 18 ++- forgelm/config.py | 29 +++- forgelm/verify/_annex_iv.py | 7 +- forgelm/wizard/__init__.py | 2 +- forgelm/wizard/_collectors.py | 22 ++- site/js/translations.js | 88 ++++++----- site/js/wizard.js | 72 +++++++-- tests/site/wizard_monitoring.test.cjs | 145 ++++++++++++++++++ tests/test_config.py | 32 +++- tests/test_eu_ai_act.py | 45 +++++- tests/test_guard_wiring.py | 1 + tests/test_wizard_phase22.py | 18 +-- tools/check_module_size.py | 6 +- 24 files changed, 575 insertions(+), 118 deletions(-) create mode 100644 tests/site/wizard_monitoring.test.cjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9bface50..3e547232 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -161,6 +161,11 @@ jobs: steps: - uses: actions/checkout@v5 + - uses: actions/setup-node@v7 + with: + node-version: "24" + package-manager-cache: false + - uses: actions/setup-python@v6 with: python-version: "3.11" @@ -181,6 +186,9 @@ jobs: - name: Import-origin guard run: python3 tools/check_import_origin.py --strict + - name: Browser wizard regression tests + run: node --test tests/site/*.test.cjs + - name: CLI smoke checks run: | forgelm --version diff --git a/AGENTS.md b/AGENTS.md index 629b12a7..d3b0470b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -166,6 +166,7 @@ Default workflow for a non-trivial change: ```bash python3 tools/check_import_origin.py --strict && \ ruff format . && ruff check . && pytest tests/ && \ + node --test tests/site/*.test.cjs && \ python3 -m forgelm --config config_template.yaml --dry-run && \ python3 -m mypy --strict --follow-imports=silent forgelm/__init__.py forgelm/_version.py tests/typing/public_surface_probe.py && \ python3 tools/check_field_descriptions.py --strict forgelm/config.py && \ diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d620f6e..ad44fee6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,21 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Breaking + +- Tighten `monitoring.endpoint_env` to trimmed ASCII identifiers; URLs, `$`/`env:` prefixes, + internal whitespace, and non-ASCII names now fail validation. Blank names remain absent. + This changes acceptance of previously valid YAML; use a variable name, such as `MONITORING_PLAN_URL`. + ### Changed +- Record monitoring default provenance in `defaulted_fields` and mark reduced literal URLs with + `endpoint_redacted`. Literal endpoints now retain only scheme/host in evidence; env references + remain unresolved. Empty endpoints record `null`, and non-empty `endpoint_env` takes precedence. + Monitoring declarations remain optional in ForgeLM at every risk tier, including strict tiers; + providers’ Article 72 obligations apply independently. Treat legacy monitoring blocks without + `declarative_only` as declarations; their default provenance was not recorded (#93). + - Redesign the root README around a GPU-free first run, a concise capability overview, optional evidence controls, and clear contribution and documentation paths. Reuse the site's ForgeLM mark and distinguish the Apache-licensed code from CC-BY-SA seed datasets. @@ -16,11 +29,13 @@ All notable changes to ForgeLM are documented here. ### Fixed - Restore the monitoring-plan evidence contract: `monitoring.*` declares an external plan, - labelled `declarative_only: true` in the compliance manifest and Annex IV; ForgeLM runs no - exporter, drift detector, alert, or scheduler. Warn when an enabled plan names an exporter, - record absent endpoints as `null`, and honor unresolved `endpoint_env` references over literal - endpoints. Align CLI/web wizards, six site locales, and EN/TR references and QMS guidance; - correct the site's claim that webhook delivery attempts are recorded in the audit log (#93). + labelled `declarative_only: true`; ForgeLM runs no exporter, drift detector, alert, or scheduler. + Warn once per config-file load for an enabled exporter declaration, without repeating during + pipeline stage validation. Align CLI/web wizards and EN/TR guidance, correct Annex IV monitoring + point 9 / risk-management point 5 labels, and verify exported declarations end to end (#93). +- Correct webhook troubleshooting in the site and EN/TR manuals: delivery failures appear as + `forgelm.webhook` warnings, never `webhook_failed` audit events; use the actual + `allow_private_destinations` setting (#231). - Rework the homepage terminal into a first-run guide with separately copyable installation and config-generation commands, an optional training step, and readable code in both themes and on narrow screens, with interface copy in all six site languages. Keep the dated CPU recording and diff --git a/CLAUDE.md b/CLAUDE.md index e293ab9c..67197574 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -166,6 +166,7 @@ Default workflow for a non-trivial change: ```bash python3 tools/check_import_origin.py --strict && \ ruff format . && ruff check . && pytest tests/ && \ + node --test tests/site/*.test.cjs && \ python3 -m forgelm --config config_template.yaml --dry-run && \ python3 -m mypy --strict --follow-imports=silent forgelm/__init__.py forgelm/_version.py tests/typing/public_surface_probe.py && \ python3 tools/check_field_descriptions.py --strict forgelm/config.py && \ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 900a46a0..165b3c77 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -165,6 +165,7 @@ supported Python matrix: ```bash python3 tools/check_import_origin.py --strict && \ ruff format . && ruff check . && pytest tests/ && \ + node --test tests/site/*.test.cjs && \ python3 -m forgelm --config config_template.yaml --dry-run && \ python3 -m mypy --strict --follow-imports=silent forgelm/__init__.py forgelm/_version.py tests/typing/public_surface_probe.py && \ python3 tools/check_field_descriptions.py --strict forgelm/config.py && \ diff --git a/config_template.yaml b/config_template.yaml index 50e7b499..86dbed3a 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -261,8 +261,9 @@ data: # export metrics, detect drift, send drift alerts, or schedule checks. Operate your own stack. # monitoring: # enabled: true -# endpoint_env: "MONITORING_WEBHOOK_URL" # recorded as "$NAME", never read; overrides endpoint -# metrics_export: "prometheus" # "none", "prometheus", "datadog", "custom_webhook" +# endpoint: "" # optional literal URL; evidence keeps scheme/host only +# endpoint_env: "MONITORING_PLAN_URL" # recorded as "$NAME", never read; overrides endpoint +# metrics_export: "none" # other exporters emit a metadata-only warning at config load # alert_on_drift: true # check_interval_hours: 24 diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index 689add6d..93da2c2b 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -320,24 +320,52 @@ uzatmasını engeller. Bu blok **yalnızca beyan niteliğinde metadata** içerir. `enabled: true` olduğunda ForgeLM, harici pazar sonrası izleme planınızı uyumluluk manifestine ve Annex IV `monitoring_and_logging` bölümüne -`declarative_only: true` etiketiyle kaydeder. Endpoint'e bağlanmaz, metrik aktarmaz, model sapması -tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. İzleme altyapınızı ayrıca işletin; +`declarative_only: true` etiketiyle kaydeder. Endpoint'e bağlanmaz, metrikleri dışa aktarmaz, drift +tespit etmez, drift uyarısı göndermez veya kontrol zamanlamaz. İzleme altyapınızı ayrıca işletin; bu beyan izlemenin çalıştığını kanıtlamaz ve izleme yükümlülüğünü tek başına karşılamaz. -Pazar sonrası izleme planı [Madde 72](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-72), -QMS ise [Madde 17(1)(h)](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-17) kapsamındadır. +Pazar sonrası izleme planı [Madde 72](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng#art_72), +QMS ise [Madde 17(1)(h)](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng#art_17) kapsamındadır. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `enabled` | bool | `false` | Harici planı manifeste kaydet; tüm risk seviyelerinde isteğe bağlıdır | -| `endpoint` | string | `""` | Beyan edilen harici izleme URL'si; isteğe bağlıdır, bağlantı kurulmaz | -| `endpoint_env` | string | `null` | Boş olmayan değişken adı `endpoint`'ten önceliklidir; `$NAME` olarak kaydedilir, değeri okunmaz | +| `enabled` | bool | `false` | Harici planı manifeste kaydet; metadata kaydı isteğe bağlıdır; yasal yükümlülükler ayrıca geçerlidir | +| `endpoint` | string | `""` | İsteğe bağlı URL; delile yalnızca scheme/host yazılır, bağlantı kurulmaz | +| `endpoint_env` | string | `null` | ASCII tanımlayıcısı `endpoint`’ten önceliklidir; düz metin delile `$NAME` yazılır, değeri okunmaz; kimlik bilgisi yazmayın | | `metrics_export` | string | `"none"` | Beyan edilen harici aktarım sistemi: `"none"`, `"prometheus"`, `"datadog"`, `"custom_webhook"`; etkin planda `none` dışındaki seçim yapılandırma yüklenirken uyarı üretir | -| `alert_on_drift` | bool | `true` | Harici planda sapma uyarısı öngörülüp öngörülmediği; uyarıları etkinleştirmez | +| `alert_on_drift` | bool | `true` | Harici planda drift uyarısı öngörülüp öngörülmediği; uyarıları etkinleştirmez | | `check_interval_hours` | int | `24` | Beyan edilen harici kontrol aralığı, saat (en az 1); kontrol zamanlamaz | Her iki endpoint alanı da boşsa manifeste `endpoint: null` yazılır. Adlandırılan ortam değişkeni tanımlı olsa bile referansı çözümlenmez. Kapalı veya belirtilmemiş planlar kaydedilmez. +ForgeLM’de metadata kaydı isteğe bağlıdır; Madde 72, yüksek riskli yapay zekâ sistemlerinin sağlayıcılarının pazar sonrası izleme sistemi kurmasını ve belgelemesini gerektirir. Bu alanlar bir özettir; Annex IV madde 9’un (Madde 72(3)) gerektirdiği ayrıntılı plan değildir. + +Delil dosyaları (`compliance_report.json`, `training_manifest.yaml` ve üretildiğinde `annex_iv_metadata.json`) düz metindir. Literal `endpoint` kırpılır ve yalnızca scheme/host kaydedilir; kullanıcı bilgisi, port, path, query ve fragment çıkarılır, `endpoint_redacted: true` bu azaltmayı gösterir. İki alana da kimlik bilgisi yazmayın; sır içeren URL için `endpoint_env` kullanın. Bu adın çevresindeki boşluklar kırpılır; ASCII tanımlayıcısı olmalıdır (önce harf veya alt çizgi, ardından harf, rakam veya alt çizgi). Değişkenin değeri okunmadan adı `$NAME` olarak aynen kaydedilir. Boş ad literal endpoint’e geri döner. + +`defaulted_fields`, operatörün açıkça yazdığı değerler yerine şema varsayılanlarından gelen kayıt alanlarını listeler. Yalnızca planı etkinleştirmek, 24 saatlik kontrol aralığı veya drift uyarısı politikası beyan etmez. Örneğin `monitoring: {enabled: true}`, `compliance_report.json` içinde şu kaydı üretir: + +```json +{ + "monitoring": { + "declarative_only": true, + "endpoint": null, + "endpoint_redacted": false, + "metrics_export": "none", + "alert_on_drift": true, + "check_interval_hours": 24, + "defaulted_fields": [ + "endpoint", + "metrics_export", + "alert_on_drift", + "check_interval_hours" + ] + } +} +``` + +Endpoint yoksa delile `null` yazılır; girdi YAML’ında `endpoint` alanını atlayın veya `endpoint: ""` kullanın (`null` değil). `compliance:` içindeki sağlayıcı adı, sistem adı veya kullanım amacından en az biri doluysa Annex IV dosyası üretilir; doğrulama üçünün de dolu olmasını gerektirir. `declarative_only` anahtarı bulunmayan eski izleme kayıtlarını da beyan olarak değerlendirin; çalışan izleme kanıtı saymayın. Eski delilde `defaulted_fields` yoksa varsayılanların kaynağı kaydedilmemiştir. + + ## `distributed` (İsteğe bağlı) | Alan | Tip | Varsayılan | Açıklama | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index cf52cc69..f5b87f41 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -381,14 +381,14 @@ post-market monitoring plan in the compliance manifest and Annex IV `monitoring_ labelled `declarative_only: true`. It does not contact the endpoint, export metrics, detect drift, send drift alerts, or schedule checks. Operate your monitoring stack separately; this declaration does not establish that monitoring is running or satisfy the monitoring obligation by itself. -The post-market monitoring plan is covered by [Article 72](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-72) -and the QMS by [Article 17(1)(h)](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-17). +The post-market monitoring plan is covered by [Article 72](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng#art_72) +and the QMS by [Article 17(1)(h)](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng#art_17). | Field | Type | Default | Description | |-------|------|---------|-------------| -| `enabled` | bool | `false` | Record the external plan in the manifest; opt-in for every risk tier | -| `endpoint` | string | `""` | Declared external monitoring URL; optional, never contacted | -| `endpoint_env` | string | `null` | Non-empty variable name takes precedence over `endpoint`; recorded as `$NAME`, never read | +| `enabled` | bool | `false` | Record the external plan in the manifest; optional metadata recording; legal obligations apply separately | +| `endpoint` | string | `""` | Optional URL; evidence keeps scheme/host only, never contacted | +| `endpoint_env` | string | `null` | ASCII identifier takes precedence over `endpoint`; recorded as `$NAME` in plain-text evidence, never read; no credentials | | `metrics_export` | string | `"none"` | Declared external exporter: `"none"`, `"prometheus"`, `"datadog"`, `"custom_webhook"`; an enabled non-`none` declaration emits a config-load warning | | `alert_on_drift` | bool | `true` | Whether the external plan calls for drift alerts; does not activate alerts | | `check_interval_hours` | int | `24` | Declared external cadence in hours (minimum 1); does not schedule checks | @@ -396,6 +396,33 @@ and the QMS by [Article 17(1)(h)](https://ai-act-service-desk.ec.europa.eu/en/ai If both endpoint fields are empty, the manifest records `endpoint: null`. Environment-variable references remain unresolved even if the named variable is set. Disabled or omitted plans are not recorded. +Recording metadata in ForgeLM is optional; Article 72 requires providers of high-risk AI systems to establish and document a post-market monitoring system. These fields are a summary, not the detailed plan required by Annex IV point 9 (Article 72(3)). + +Evidence files (`compliance_report.json`, `training_manifest.yaml`, and, when emitted, `annex_iv_metadata.json`) are plain text. A literal `endpoint` is trimmed and reduced to scheme/host; userinfo, port, path, query, and fragment are removed and `endpoint_redacted: true` marks that reduction. Do not put credentials in either field; use `endpoint_env` for a secret-bearing URL. Its name is trimmed, must be an ASCII identifier (letters or underscore first, then letters, digits, or underscores), and is recorded verbatim as `$NAME` without reading the variable. A blank name falls back to the literal endpoint. + +`defaulted_fields` lists recorded fields whose values came from schema defaults, rather than explicit operator input. An enabled plan alone therefore does not declare a 24-hour check cadence or a drift-alert policy. For example, `monitoring: {enabled: true}` produces this entry in `compliance_report.json`: + +```json +{ + "monitoring": { + "declarative_only": true, + "endpoint": null, + "endpoint_redacted": false, + "metrics_export": "none", + "alert_on_drift": true, + "check_interval_hours": 24, + "defaulted_fields": [ + "endpoint", + "metrics_export", + "alert_on_drift", + "check_interval_hours" + ] + } +} +``` + +An absent endpoint is `null` in evidence; in input YAML omit `endpoint` or use `endpoint: ""` (not `null`). Annex IV output is emitted when `compliance:` contains at least one non-blank provider name, system name, or intended purpose; verification requires all three. Legacy monitoring blocks without `declarative_only` must also be treated as declarations, never proof of runtime monitoring. Missing `defaulted_fields` in old evidence means the provenance of defaults was not recorded. + --- ## `synthetic` (Optional — Synthetic Data Generation) diff --git a/docs/standards/testing.md b/docs/standards/testing.md index 12e14948..70bfc53e 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -61,6 +61,14 @@ tests/ ## Fixtures +Browser wizard regressions live in `tests/site/*.test.cjs`. Run +`node --test tests/site/*.test.cjs` with Node 22 or newer and the development +Python environment activated. CI runs them on Node 24 in the validation job. +They use Node's built-in test runner and a VM to execute the actual wizard; +generated YAML is also parsed by the real `ForgeConfig`. No npm packages, +browser download, GPU, or network calls are needed. DOM shims exercise the +Operations renderer only; they do not replace browser or native-language review. + The canonical factory lives in [`tests/_helpers/factories.py`](../../tests/_helpers/factories.py) and is re-exported via [`tests/conftest.py`](../../tests/conftest.py) (both as the diff --git a/docs/usermanuals/en/reference/configuration.md b/docs/usermanuals/en/reference/configuration.md index e52296e4..e139d050 100644 --- a/docs/usermanuals/en/reference/configuration.md +++ b/docs/usermanuals/en/reference/configuration.md @@ -295,8 +295,8 @@ proof that monitoring is running. The block remains optional for every risk tier ```yaml monitoring: enabled: false # Record external plan metadata only - endpoint: "" # Optional external URL; never contacted - endpoint_env: null # Non-empty name overrides endpoint as "$NAME"; never read + endpoint: "" # Optional URL; evidence keeps scheme/host only + endpoint_env: null # ASCII name overrides endpoint as "$NAME"; never read metrics_export: "none" # Declared only: none | prometheus | datadog | custom_webhook alert_on_drift: true # Declared external drift-alert policy; no alerts sent check_interval_hours: 24 # Declared external cadence; no checks scheduled @@ -306,6 +306,34 @@ An enabled non-`none` exporter declaration emits a config-load warning. If neith is set, the manifest endpoint is `null`. Environment references are recorded by name even when the variable is set; disabled or omitted plans are not recorded. +Recording metadata in ForgeLM is optional; Article 72 requires providers of high-risk AI systems to establish and document a post-market monitoring system. These fields are a summary, not the detailed plan required by Annex IV point 9 (Article 72(3)). + +Evidence files (`compliance_report.json`, `training_manifest.yaml`, and, when emitted, `annex_iv_metadata.json`) are plain text. A literal `endpoint` is trimmed and reduced to scheme/host; userinfo, port, path, query, and fragment are removed and `endpoint_redacted: true` marks that reduction. Do not put credentials in either field; use `endpoint_env` for a secret-bearing URL. Its name is trimmed, must be an ASCII identifier (letters or underscore first, then letters, digits, or underscores), and is recorded verbatim as `$NAME` without reading the variable. A blank name falls back to the literal endpoint. + +`defaulted_fields` lists recorded fields whose values came from schema defaults, rather than explicit operator input. An enabled plan alone therefore does not declare a 24-hour check cadence or a drift-alert policy. For example, `monitoring: {enabled: true}` produces this entry in `compliance_report.json`: + +```json +{ + "monitoring": { + "declarative_only": true, + "endpoint": null, + "endpoint_redacted": false, + "metrics_export": "none", + "alert_on_drift": true, + "check_interval_hours": 24, + "defaulted_fields": [ + "endpoint", + "metrics_export", + "alert_on_drift", + "check_interval_hours" + ] + } +} +``` + +An absent endpoint is `null` in evidence; in input YAML omit `endpoint` or use `endpoint: ""` (not `null`). Annex IV output is emitted when `compliance:` contains at least one non-blank provider name, system name, or intended purpose; verification requires all three. Legacy monitoring blocks without `declarative_only` must also be treated as declarations, never proof of runtime monitoring. Missing `defaulted_fields` in old evidence means the provenance of defaults was not recorded. + + ## `retention:` ```yaml diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 6d7e3923..35d7c90d 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -290,24 +290,52 @@ risk_assessment: Bu blok harici bir pazar sonrası izleme planını beyan eder (Madde 72 / 17(1)(h)). `enabled: true` olduğunda değerleri uyumluluk manifestine ve Annex IV `monitoring_and_logging` bölümüne -`declarative_only: true` etiketiyle kaydedilir. ForgeLM endpoint'e bağlanmaz, metrik aktarmaz, -model sapması tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. İzleme altyapınızı ayrıca -işletin; bu beyan izlemenin çalıştığını kanıtlamaz. Blok tüm risk seviyelerinde isteğe bağlıdır. +`declarative_only: true` etiketiyle kaydedilir. ForgeLM endpoint'e bağlanmaz, metrikleri dışa aktarmaz, +drift tespit etmez, drift uyarısı göndermez veya kontrol zamanlamaz. İzleme altyapınızı ayrıca +işletin; bu beyan izlemenin çalıştığını kanıtlamaz. ForgeLM’de metadata kaydı isteğe bağlıdır; yasal yükümlülükler ayrıca geçerlidir. ```yaml monitoring: - enabled: false # Record external plan metadata only - endpoint: "" # Optional external URL; never contacted - endpoint_env: null # Non-empty name overrides endpoint as "$NAME"; never read - metrics_export: "none" # Declared only: none | prometheus | datadog | custom_webhook - alert_on_drift: true # Declared external drift-alert policy; no alerts sent - check_interval_hours: 24 # Declared external cadence; no checks scheduled + enabled: false # Yalnızca harici plan metadata’sını kaydet + endpoint: "" # İsteğe bağlı URL; delilde yalnızca scheme/host + endpoint_env: null # ASCII ad endpoint’ten öncelikli; "$NAME" yazılır, okunmaz + metrics_export: "none" # Yalnızca beyan: none | prometheus | datadog | custom_webhook + alert_on_drift: true # Harici drift uyarısı politikası; uyarı gönderilmez + check_interval_hours: 24 # Harici kontrol aralığı; kontrol zamanlanmaz ``` Etkin planda `none` dışındaki aktarım sistemi seçimi yapılandırma yüklenirken uyarı üretir. Her iki endpoint alanı da boşsa manifeste `null` yazılır. Ortam değişkeni tanımlı olsa bile yalnızca adı kaydedilir; kapalı veya belirtilmemiş planlar kaydedilmez. +ForgeLM’de metadata kaydı isteğe bağlıdır; Madde 72, yüksek riskli yapay zekâ sistemlerinin sağlayıcılarının pazar sonrası izleme sistemi kurmasını ve belgelemesini gerektirir. Bu alanlar bir özettir; Annex IV madde 9’un (Madde 72(3)) gerektirdiği ayrıntılı plan değildir. + +Delil dosyaları (`compliance_report.json`, `training_manifest.yaml` ve üretildiğinde `annex_iv_metadata.json`) düz metindir. Literal `endpoint` kırpılır ve yalnızca scheme/host kaydedilir; kullanıcı bilgisi, port, path, query ve fragment çıkarılır, `endpoint_redacted: true` bu azaltmayı gösterir. İki alana da kimlik bilgisi yazmayın; sır içeren URL için `endpoint_env` kullanın. Bu adın çevresindeki boşluklar kırpılır; ASCII tanımlayıcısı olmalıdır (önce harf veya alt çizgi, ardından harf, rakam veya alt çizgi). Değişkenin değeri okunmadan adı `$NAME` olarak aynen kaydedilir. Boş ad literal endpoint’e geri döner. + +`defaulted_fields`, operatörün açıkça yazdığı değerler yerine şema varsayılanlarından gelen kayıt alanlarını listeler. Yalnızca planı etkinleştirmek, 24 saatlik kontrol aralığı veya drift uyarısı politikası beyan etmez. Örneğin `monitoring: {enabled: true}`, `compliance_report.json` içinde şu kaydı üretir: + +```json +{ + "monitoring": { + "declarative_only": true, + "endpoint": null, + "endpoint_redacted": false, + "metrics_export": "none", + "alert_on_drift": true, + "check_interval_hours": 24, + "defaulted_fields": [ + "endpoint", + "metrics_export", + "alert_on_drift", + "check_interval_hours" + ] + } +} +``` + +Endpoint yoksa delile `null` yazılır; girdi YAML’ında `endpoint` alanını atlayın veya `endpoint: ""` kullanın (`null` değil). `compliance:` içindeki sağlayıcı adı, sistem adı veya kullanım amacından en az biri doluysa Annex IV dosyası üretilir; doğrulama üçünün de dolu olmasını gerektirir. `declarative_only` anahtarı bulunmayan eski izleme kayıtlarını da beyan olarak değerlendirin; çalışan izleme kanıtı saymayın. Eski delilde `defaulted_fields` yoksa varsayılanların kaynağı kaydedilmemiştir. + + ## `retention:` ```yaml diff --git a/forgelm/compliance.py b/forgelm/compliance.py index 4861decf..1772714f 100644 --- a/forgelm/compliance.py +++ b/forgelm/compliance.py @@ -20,6 +20,7 @@ from datetime import datetime, timezone from typing import Any, Dict, List, Optional, Set, Tuple +from ._http import _mask_netloc from ._strict_json import dumps_strict, sanitize_non_finite from ._version import __version__ as _forgelm_version from .config import ConfigError, WebhookConfig @@ -1583,10 +1584,21 @@ def generate_training_manifest( if mon_cfg and mon_cfg.enabled: manifest["monitoring"] = { "declarative_only": True, - "endpoint": f"${mon_cfg.endpoint_env}" if mon_cfg.endpoint_env else (mon_cfg.endpoint or None), + "endpoint": ( + f"${mon_cfg.endpoint_env}" + if mon_cfg.endpoint_env + else (_mask_netloc(mon_cfg.endpoint) if mon_cfg.endpoint else None) + ), + "endpoint_redacted": bool(mon_cfg.endpoint and not mon_cfg.endpoint_env), "metrics_export": mon_cfg.metrics_export, "alert_on_drift": mon_cfg.alert_on_drift, "check_interval_hours": mon_cfg.check_interval_hours, + "defaulted_fields": [ + name + for name in ("endpoint", "metrics_export", "alert_on_drift", "check_interval_hours") + if ("endpoint_env" if name == "endpoint" and mon_cfg.endpoint_env else name) + not in mon_cfg.model_fields_set + ], } # Webhook config — preserved into the compliance report so the @@ -1854,11 +1866,11 @@ def build_annex_iv_artifact(manifest: Dict[str, Any]) -> Optional[Dict[str, Any] "generated_at": manifest.get("generated_at", ""), "known_limitations": operator_block.get("known_limitations", ""), }, - # Declared external monitoring metadata, or the audit-log reference. + # Annex IV point 9 (Art. 72(3)): external plan metadata, not the full plan. "monitoring_and_logging": (manifest.get("monitoring") or {"audit_log": "audit_log.jsonl"}), # Annex IV §7: accuracy / robustness metrics. "performance_metrics": manifest.get("evaluation_results", {}).get("metrics", {}), - # Annex IV §9: risk management system reference. + # Annex IV point 5: risk management system reference. "risk_management": manifest.get("risk_assessment") or { "art9_reference": "no risk_assessment block configured", diff --git a/forgelm/config.py b/forgelm/config.py index 9e601d48..e415f636 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1296,11 +1296,12 @@ class MonitoringConfig(BaseModel): default=False, description="Record external monitoring-plan metadata in the compliance manifest." ) endpoint: str = Field( - default="", description="Declared external monitoring URL; recorded only, never contacted by ForgeLM." + default="", + description="External plan URL; evidence records only scheme/host. Never contacted. Use endpoint_env for credentials.", ) endpoint_env: Optional[str] = Field( default=None, - description="Env var name recorded as `$NAME`, overriding `endpoint` when non-empty; never resolved.", + description="ASCII env name recorded as `$NAME` in plain-JSON evidence, overriding endpoint; never resolved. No credentials.", ) metrics_export: Literal["none", "prometheus", "datadog", "custom_webhook"] = Field( default="none", @@ -1314,15 +1315,31 @@ class MonitoringConfig(BaseModel): default=24, ge=1, description="Declared external check cadence in hours; ForgeLM does not schedule checks." ) - @model_validator(mode="after") - def _warn_declarative_exporter(self) -> "MonitoringConfig": + @field_validator("endpoint") + @classmethod + def _strip_endpoint(cls, value: str) -> str: + """Normalize an optional literal URL before its evidence projection.""" + return value.strip() + + @field_validator("endpoint_env") + @classmethod + def _validate_endpoint_env(cls, value: Optional[str]) -> Optional[str]: + """Accept an unresolved ASCII identifier, with surrounding whitespace removed.""" + if value is None or not value.strip(): + return None + value = value.strip() + if not value.isascii() or not value.isidentifier(): + raise ValueError("monitoring.endpoint_env must be an ASCII environment-variable name, without env: or $.") + return value + + def _warn_declarative_exporter(self) -> None: + """Warn once at the file-load boundary, never during pipeline stage validation.""" if self.enabled and self.metrics_export != "none": logger.warning( "monitoring.metrics_export is declarative metadata only: ForgeLM does not export metrics, " "contact the endpoint, detect drift, send drift alerts, or schedule checks. " "Operate your external monitoring stack separately." ) - return self class ComplianceMetadataConfig(BaseModel): @@ -2295,4 +2312,6 @@ def load_config(config_path: str) -> ForgeConfig: # / TypeError outside Pydantic's wrapper. Same message shape. raise ConfigError(f"Configuration validation failed: {e}") from e + if config.monitoring is not None: + config.monitoring._warn_declarative_exporter() return config diff --git a/forgelm/verify/_annex_iv.py b/forgelm/verify/_annex_iv.py index 52e97f14..352a5146 100644 --- a/forgelm/verify/_annex_iv.py +++ b/forgelm/verify/_annex_iv.py @@ -38,9 +38,12 @@ ("computational_resources", "Annex IV §2(g) — compute resources used during training"), ("data_governance", "Annex IV §2(d) — data sources, governance, validation methodology"), ("technical_documentation", "Annex IV §3-5 — design + development methodology"), - ("monitoring_and_logging", "Annex IV §6 — post-market monitoring + audit-log presence"), + ( + "monitoring_and_logging", + "Annex IV point 9 (Art. 72(3)) — declared external plan metadata or audit-log reference", + ), ("performance_metrics", "Annex IV §7 — accuracy / robustness / cybersecurity metrics"), - ("risk_management", "Annex IV §9 — risk management system reference (Art. 9 alignment)"), + ("risk_management", "Annex IV point 5 — risk management system reference (Art. 9 alignment)"), ) diff --git a/forgelm/wizard/__init__.py b/forgelm/wizard/__init__.py index 3e8fc4ca..8d9e8b36 100644 --- a/forgelm/wizard/__init__.py +++ b/forgelm/wizard/__init__.py @@ -7,7 +7,7 @@ hyperparameters (``dpo_beta``, ``simpo_beta`` / ``simpo_gamma``, ``kto_beta``, ``orpo_beta``, ``grpo_*``), full PEFT method coverage (``lora`` / ``dora`` / ``pissa`` / ``rslora``) plus GaLore as a -separate axis, EU AI Act Article 9 / 10 / 11 / 12+17 compliance +separate axis, EU AI Act Article 9 / 10 / 11 / 12 / 17(1)(h) / 72 compliance accordions, F-compliance-110 strict-tier auto-coercion, ``back`` / ``reset`` navigation, XDG-aware persistence at ``$XDG_CACHE_HOME/forgelm/wizard_state.yaml``, step-diff preview, diff --git a/forgelm/wizard/_collectors.py b/forgelm/wizard/_collectors.py index f1dfee6b..a39c4a2d 100644 --- a/forgelm/wizard/_collectors.py +++ b/forgelm/wizard/_collectors.py @@ -10,7 +10,6 @@ from __future__ import annotations -import re from dataclasses import dataclass from pathlib import Path from typing import Any, Dict, Optional, Tuple @@ -90,15 +89,11 @@ def _default_safety_probes_path() -> str: # --------------------------------------------------------------------------- # Shared ``env:VAR_NAME`` reference parsing — the webhook URL prompt and # the monitoring-endpoint prompt both accept this indirection syntax. -# Centralised here so the accepted env-var-name shape (POSIX: uppercase -# letters / digits / underscores, must start with a letter) only needs +# Centralised here so the accepted ASCII-identifier shape only needs # updating in one place. # --------------------------------------------------------------------------- -_ENV_VAR_NAME_RE = re.compile(r"^[A-Z][A-Z0-9_]*$", flags=re.ASCII) - - def _parse_env_ref(raw: str) -> Optional[str]: """Parse an ``env:VAR_NAME`` reference; return the variable name. @@ -116,10 +111,9 @@ def _parse_env_ref(raw: str) -> Optional[str]: var_name = raw[4:].strip() if not var_name: raise ValueError("`env:` prefix needs a non-empty variable name (e.g. `env:SLACK_WEBHOOK_URL`).") - if not _ENV_VAR_NAME_RE.match(var_name): + if not var_name.isascii() or not var_name.isidentifier(): raise ValueError( - f"`env:{var_name}` is not a POSIX environment-variable name " - "(uppercase letters / digits / underscores, must start with a letter)." + "Not a POSIX environment-variable name: use ASCII letters, digits, or underscores; no initial digit." ) return var_name @@ -887,7 +881,7 @@ def _collect_monitoring() -> Optional[Dict[str, Any]]: return None _print(" Metadata only: ForgeLM does not export metrics, detect drift, send drift alerts, or schedule checks.") endpoint_raw = _prompt( - "Declared monitoring URL (optional; `env:VAR_NAME` is recorded without reading the variable)", + "External plan URL (optional; literal evidence keeps scheme/host only; prefer `env:VAR_NAME` for secrets)", "", ) metrics_export = _prompt_choice( @@ -898,8 +892,10 @@ def _collect_monitoring() -> Optional[Dict[str, Any]]: monitoring: Dict[str, Any] = { "enabled": True, "metrics_export": metrics_export, - "alert_on_drift": _prompt_yes_no("External monitoring plan: alert_on_drift?", default=True), - "check_interval_hours": _prompt_int("Declared external check_interval_hours", 24, min_val=1, max_val=720), + "alert_on_drift": _prompt_yes_no("Does your external monitoring stack alert on drift?", default=True), + "check_interval_hours": _prompt_int( + "How often does your external stack check the model (hours)?", 24, min_val=1, max_val=720 + ), } # P9: support env:VAR_NAME indirection for the monitoring endpoint # the same way the webhook collector does. ``MonitoringConfig`` has @@ -910,7 +906,7 @@ def _collect_monitoring() -> Optional[Dict[str, Any]]: try: monitoring["endpoint_env"] = _parse_env_ref(endpoint_stripped) except ValueError: - _print(f" '{endpoint_stripped}' is not a valid env-var reference — monitoring endpoint left unset.") + _print(" Input is not a valid env-var reference — monitoring endpoint left unset.") elif endpoint_stripped: monitoring["endpoint"] = endpoint_stripped return monitoring diff --git a/site/js/translations.js b/site/js/translations.js index 3017a2b9..840d2067 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -1677,8 +1677,8 @@ "wizard.step.compliance.tutorial": "ForgeLM checks the fine-tuned model BEFORE you ship it. Auto-revert undoes the fine-tune if it's worse than the base. Llama Guard scans outputs for harmful content. The EU AI Act risk tier you pick determines which compliance artefacts get auto-generated and whether human approval is REQUIRED before deployment.", "wizard.step.operations.title": "Operations & advanced settings", "wizard.step.operations.name": "Operations", - "wizard.step.operations.desc": "Notifications, post-market monitoring, synthetic data generation, and retention horizons. All optional — skip the ones you don't need.", - "wizard.step.operations.tutorial": "These four blocks cover everything ForgeLM ships beyond the core training + compliance flow. Webhook posts a Slack/Teams/Discord alert when training starts, succeeds, or fails. Monitoring (Art. 72 / 17(1)(h)) records an external plan as metadata; ForgeLM does not export metrics, detect drift, send drift alerts, or schedule checks. Synthetic data generation lets a stronger teacher model produce extra training examples. Retention horizons set GDPR-compliant TTLs for the audit log and intermediate artefacts. None of these are required — leave them collapsed if your run doesn't need them.", + "wizard.step.operations.desc": "Notifications, external monitoring-plan metadata, synthetic data, and retention settings. Configure the ForgeLM options you need.", + "wizard.step.operations.tutorial": "These four blocks cover everything ForgeLM ships beyond the core training + compliance flow. Webhook posts a Slack/Teams/Discord alert when training starts, succeeds, or fails. Monitoring (Art. 72 / 17(1)(h)) records an external plan as metadata; ForgeLM does not export metrics, detect drift, send drift alerts, or schedule checks. Synthetic data generation lets a stronger teacher model produce extra training examples. Retention horizons set GDPR-compliant TTLs for the audit log and intermediate artefacts. These ForgeLM settings are optional; external legal obligations still apply.", "wizard.risk.label": "EU AI Act risk classification", "wizard.risk.hint": "Pick the tier your deployment falls under. High-risk and unacceptable trigger mandatory audit + human-approval obligations.", "wizard.risk.unknown": "Unknown — pre-classification placeholder", @@ -1782,10 +1782,10 @@ "wizard.monitoring.title": "External monitoring plan (Art. 72 / 17(1)(h))", "wizard.monitoring.hint": "Declare your external monitoring setup in the compliance manifest / Annex IV. Metadata only; ForgeLM runs no monitoring.", "wizard.monitoring.enabled.label": "Record monitoring-plan metadata", - "wizard.monitoring.enabled.hint": "Optional for every risk tier. Recording a plan does not establish that monitoring is running.", + "wizard.monitoring.enabled.hint": "Recording metadata in ForgeLM is optional. Art. 72 requires providers of high-risk AI systems to operate a post-market monitoring system.", "wizard.monitoring.enabled.detail": "ForgeLM records the declared plan with declarative_only: true. It does not contact the endpoint, export metrics, detect drift, send drift alerts, or schedule checks. Operate your external stack separately; this declaration alone does not satisfy a post-market monitoring obligation.", "wizard.monitoring.endpoint.label": "Declared endpoint (optional)", - "wizard.monitoring.endpoint.hint": "Recorded only, never contacted. env:NAME is stored as $NAME without reading the variable; no endpoint records null.", + "wizard.monitoring.endpoint.hint": "Never contacted. Plain-text evidence keeps only scheme/host for literal URLs; no credentials. Prefer env:NAME: only $NAME is recorded, never its value.", "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com OR env:MONITORING_URL", "wizard.monitoring.metrics_export.label": "Declared external exporter", "wizard.monitoring.metrics_export.none": "None", @@ -1794,6 +1794,9 @@ "wizard.monitoring.metrics_export.custom": "Custom webhook", "wizard.monitoring.alert_on_drift.label": "Declare external drift-alert policy", "wizard.monitoring.alert_on_drift.hint": "Records the policy of your external stack. ForgeLM does not detect drift or send these alerts.", + "wizard.error.env_ref.invalid": "Use env:NAME with an ASCII identifier: letters or underscore first, then letters, digits, or underscores.", + "wizard.monitoring.interval.label": "External check interval (hours, 1–720)", + "wizard.error.monitoring.interval.invalid": "Enter a whole number from 1 to 720 hours.", "wizard.synthetic.title": "Synthetic data generation (advanced)", "wizard.synthetic.hint": "Generate teacher → student training data using a stronger model.", "wizard.synthetic.enabled.label": "Generate synthetic data", @@ -2071,8 +2074,8 @@ "wizard.step.compliance.tutorial": "ForgeLM, fine-tune edilmiş modeli teslim etmeden ÖNCE kontrol eder. Auto-revert, fine-tune base'den kötüyse geri alır. Llama Guard çıktılarda zararlı içerik tarar. Seçtiğin EU AI Act risk seviyesi hangi uyumluluk artefaktlarının otomatik üretileceğini ve dağıtım öncesi insan onayının ZORUNLU olup olmayacağını belirler.", "wizard.step.operations.title": "Operasyon ve gelişmiş ayarlar", "wizard.step.operations.name": "Operasyon", - "wizard.step.operations.desc": "Bildirimler, post-market monitoring, sentetik veri üretimi ve saklama ufukları. Hepsi opsiyonel — ihtiyacın olmayanları atla.", - "wizard.step.operations.tutorial": "Bu dört blok, ForgeLM'in temel eğitim + compliance akışının ötesinde sağladığı her şeyi kapsar. Webhook, eğitim başladığında / başarılı olduğunda / hata verdiğinde Slack/Teams/Discord uyarısı gönderir. İzleme (Madde 72 / 17(1)(h)) harici planı metadata olarak kaydeder; ForgeLM metrik aktarmaz, model sapması tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. Sentetik veri üretimi, daha güçlü bir teacher modelin ek eğitim örnekleri üretmesini sağlar. Saklama ufukları audit log'u ve ara artefact'ları için GDPR uyumlu TTL'ler ayarlar. Hiçbiri zorunlu değil — eğitimin ihtiyaç duymuyorsa kapalı bırak.", + "wizard.step.operations.desc": "Bildirimler, harici izleme planı metadata’sı, sentetik veri ve saklama ayarları. Gereken ForgeLM seçeneklerini yapılandırın.", + "wizard.step.operations.tutorial": "Bu dört blok, ForgeLM'in temel eğitim + compliance akışının ötesinde sağladığı her şeyi kapsar. Webhook, eğitim başladığında / başarılı olduğunda / hata verdiğinde Slack/Teams/Discord uyarısı gönderir. İzleme (Madde 72 / 17(1)(h)) harici planı metadata olarak kaydeder; ForgeLM metrikleri dışa aktarmaz, drift tespit etmez, drift uyarısı göndermez veya kontrol zamanlamaz. Sentetik veri üretimi, daha güçlü bir teacher modelin ek eğitim örnekleri üretmesini sağlar. Saklama ufukları audit log'u ve ara artefact'ları için GDPR uyumlu TTL'ler ayarlar. Bu ForgeLM ayarları isteğe bağlıdır; harici yasal yükümlülükler ayrıca geçerlidir.", "wizard.risk.label": "EU AI Act risk sınıflandırması", "wizard.risk.hint": "Dağıtımın hangi seviyeye girdiğini seç. High-risk ve unacceptable, zorunlu denetim + insan onay yükümlülüklerini tetikler.", "wizard.risk.unknown": "Unknown — ön sınıflandırma yer tutucusu", @@ -2176,18 +2179,21 @@ "wizard.monitoring.title": "Harici izleme planı (Madde 72 / 17(1)(h))", "wizard.monitoring.hint": "Harici izleme planını uyumluluk manifestine / Annex IV’e kaydet. Yalnızca metadata; ForgeLM izleme çalıştırmaz.", "wizard.monitoring.enabled.label": "İzleme planı metadatasını kaydet", - "wizard.monitoring.enabled.hint": "Tüm risk seviyelerinde isteğe bağlıdır. Planın kaydedilmesi izlemenin çalıştığını kanıtlamaz.", - "wizard.monitoring.enabled.detail": "ForgeLM beyan edilen planı declarative_only: true etiketiyle kaydeder. Endpoint’e bağlanmaz, metrik aktarmaz, model sapması tespit etmez, sapma uyarısı göndermez veya kontrol zamanlamaz. Harici altyapını ayrıca işlet; bu beyan pazar sonrası izleme yükümlülüğünü tek başına karşılamaz.", + "wizard.monitoring.enabled.hint": "Ayarın ForgeLM’de kaydı isteğe bağlıdır. Madde 72, yüksek riskli yapay zekâ sistemlerinin sağlayıcılarına pazar sonrası izleme sistemi yükümlülüğü getirir.", + "wizard.monitoring.enabled.detail": "ForgeLM beyan edilen planı declarative_only: true etiketiyle kaydeder. Endpoint’e bağlanmaz, metrikleri dışa aktarmaz, drift tespit etmez, drift uyarısı göndermez veya kontrol zamanlamaz. Harici altyapını ayrıca işlet; bu beyan pazar sonrası izleme yükümlülüğünü tek başına karşılamaz.", "wizard.monitoring.endpoint.label": "Beyan edilen endpoint (isteğe bağlı)", - "wizard.monitoring.endpoint.hint": "Yalnızca kaydedilir, bağlantı kurulmaz. env:NAME, değişken okunmadan $NAME olarak saklanır; boş endpoint null yazılır.", + "wizard.monitoring.endpoint.hint": "Bağlantı kurulmaz. Düz metin delile literal URL’nin yalnızca scheme/host kısmı yazılır; kimlik bilgisi yazmayın. env:NAME kullanın: değeri okunmaz, yalnızca $NAME kaydedilir.", "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com VEYA env:MONITORING_URL", "wizard.monitoring.metrics_export.label": "Beyan edilen harici aktarım sistemi", "wizard.monitoring.metrics_export.none": "Yok", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "Özel webhook", - "wizard.monitoring.alert_on_drift.label": "Harici sapma uyarısı politikasını beyan et", - "wizard.monitoring.alert_on_drift.hint": "Harici altyapının politikasını kaydeder. ForgeLM model sapması tespit etmez veya bu uyarıları göndermez.", + "wizard.monitoring.alert_on_drift.label": "Harici drift uyarısı politikasını beyan et", + "wizard.monitoring.alert_on_drift.hint": "Harici altyapının politikasını kaydeder. ForgeLM drift tespit etmez veya bu uyarıları göndermez.", + "wizard.error.env_ref.invalid": "env:NAME için ASCII tanımlayıcısı kullanın: önce harf veya alt çizgi, sonra harf, rakam veya alt çizgi.", + "wizard.monitoring.interval.label": "Harici kontrol aralığı (saat, 1–720)", + "wizard.error.monitoring.interval.invalid": "1–720 arasında tam sayı saat girin.", "wizard.synthetic.title": "Sentetik veri üretimi (gelişmiş)", "wizard.synthetic.hint": "Daha güçlü bir model kullanarak teacher → student eğitim verisi üret.", "wizard.synthetic.enabled.label": "Sentetik veri üret", @@ -2465,8 +2471,8 @@ "wizard.step.compliance.tutorial": "ForgeLM prüft das fine-getunte Modell, BEVOR Sie es ausliefern. Auto-Revert macht das Fine-Tune rückgängig, wenn es schlechter als das Basismodell ist. Llama Guard scannt Ausgaben auf schädliche Inhalte. Die EU-AI-Act-Risikoklasse, die Sie wählen, bestimmt, welche Compliance-Artefakte automatisch generiert werden und ob menschliche Freigabe vor dem Deployment ERFORDERLICH ist.", "wizard.step.operations.title": "Betrieb & erweiterte Einstellungen", "wizard.step.operations.name": "Betrieb", - "wizard.step.operations.desc": "Benachrichtigungen, Post-Market-Monitoring, synthetische Datengenerierung und Retentions-Horizonte. Alle optional — überspringen, was Sie nicht brauchen.", - "wizard.step.operations.tutorial": "Diese vier Blöcke decken alles ab, was ForgeLM jenseits des Kern-Training + Compliance-Flows liefert. Webhook sendet Slack/Teams/Discord-Alerts bei Trainingsstart/Erfolg/Fehler. Monitoring (Art. 72 / 17(1)(h)) erfasst einen externen Plan als Metadaten; ForgeLM exportiert keine Metriken, erkennt keine Drift, sendet keine Drift-Warnungen und plant keine Prüfungen. Synthetische Datengenerierung lässt ein stärkeres Teacher-Modell zusätzliche Trainingsbeispiele produzieren. Retentions-Horizonte setzen DSGVO-konforme TTLs für Audit-Log und ephemere Artefakte. Nichts davon ist Pflicht — eingeklappt lassen wenn Ihr Run sie nicht braucht.", + "wizard.step.operations.desc": "Benachrichtigungen, Metadaten externer Überwachungspläne, synthetische Daten und Aufbewahrung. Konfigurieren Sie die benötigten ForgeLM-Optionen.", + "wizard.step.operations.tutorial": "Diese vier Blöcke decken alles ab, was ForgeLM jenseits des Kern-Training + Compliance-Flows liefert. Webhook sendet Slack/Teams/Discord-Alerts bei Trainingsstart/Erfolg/Fehler. Monitoring (Art. 72 / 17(1)(h)) erfasst einen externen Plan als Metadaten; ForgeLM exportiert keine Metriken, erkennt keine Drift, sendet keine Drift-Warnungen und plant keine Kontrollen. Synthetische Datengenerierung lässt ein stärkeres Teacher-Modell zusätzliche Trainingsbeispiele produzieren. Retentions-Horizonte setzen DSGVO-konforme TTLs für Audit-Log und ephemere Artefakte. Diese ForgeLM-Einstellungen sind optional; externe gesetzliche Pflichten gelten weiterhin.", "wizard.risk.label": "EU-AI-Act-Risikoklassifizierung", "wizard.risk.hint": "Wählen Sie die Klasse, unter die Ihr Deployment fällt. High-risk und unacceptable lösen verpflichtende Audit + Human-Approval-Pflichten aus.", "wizard.risk.unknown": "Unknown — Vorklassifizierungs-Platzhalter", @@ -2570,10 +2576,10 @@ "wizard.monitoring.title": "Externer Monitoring-Plan (Art. 72 / 17(1)(h))", "wizard.monitoring.hint": "Externe Überwachung im Compliance-Manifest / Annex IV dokumentieren. Nur Metadaten; ForgeLM führt keine Überwachung aus.", "wizard.monitoring.enabled.label": "Monitoring-Plan als Metadaten erfassen", - "wizard.monitoring.enabled.hint": "Für jede Risikostufe optional. Ein erfasster Plan belegt keine laufende Überwachung.", - "wizard.monitoring.enabled.detail": "ForgeLM erfasst den Plan mit declarative_only: true. Es kontaktiert keinen Endpoint, exportiert keine Metriken, erkennt keine Drift, sendet keine Drift-Warnungen und plant keine Prüfungen. Betreiben Sie Ihren externen Stack separat; die Erklärung allein erfüllt keine Pflicht zur Überwachung nach dem Inverkehrbringen.", + "wizard.monitoring.enabled.hint": "Die Metadatenerfassung in ForgeLM ist optional. Art. 72 verpflichtet Anbieter von Hochrisiko-KI-Systemen zur Überwachung nach dem Inverkehrbringen.", + "wizard.monitoring.enabled.detail": "ForgeLM erfasst den Plan mit declarative_only: true. Es kontaktiert keinen Endpoint, exportiert keine Metriken, erkennt keine Drift, sendet keine Drift-Warnungen und plant keine Kontrollen. Betreiben Sie Ihren externen Stack separat; die Erklärung allein erfüllt keine Pflicht zur Überwachung nach dem Inverkehrbringen.", "wizard.monitoring.endpoint.label": "Deklarierter Endpoint (optional)", - "wizard.monitoring.endpoint.hint": "Nur erfasst, nie kontaktiert. env:NAME wird als $NAME gespeichert, ohne die Variable zu lesen; ohne Endpoint wird null erfasst.", + "wizard.monitoring.endpoint.hint": "Nie kontaktiert. Klartextbelege enthalten bei URLs nur Schema/Host; keine Zugangsdaten eingeben. env:NAME bevorzugen: nur $NAME wird erfasst, nie der Wert.", "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com ODER env:MONITORING_URL", "wizard.monitoring.metrics_export.label": "Deklarierter externer Exporter", "wizard.monitoring.metrics_export.none": "Keine", @@ -2582,6 +2588,9 @@ "wizard.monitoring.metrics_export.custom": "Custom Webhook", "wizard.monitoring.alert_on_drift.label": "Externe Drift-Warnungen deklarieren", "wizard.monitoring.alert_on_drift.hint": "Erfasst die Richtlinie Ihres externen Stacks. ForgeLM erkennt keine Drift und sendet diese Warnungen nicht.", + "wizard.error.env_ref.invalid": "env:NAME erfordert einen ASCII-Bezeichner: zuerst Buchstabe oder Unterstrich, danach auch Ziffern.", + "wizard.monitoring.interval.label": "Externes Kontrollintervall (Stunden, 1–720)", + "wizard.error.monitoring.interval.invalid": "Geben Sie eine ganze Zahl von 1 bis 720 Stunden ein.", "wizard.synthetic.title": "Synthetische Datengenerierung (erweitert)", "wizard.synthetic.hint": "Teacher → Student Trainingsdaten mit stärkerem Modell generieren.", "wizard.synthetic.enabled.label": "Synthetische Daten generieren", @@ -2859,8 +2868,8 @@ "wizard.step.compliance.tutorial": "ForgeLM vérifie le modèle fine-tuné AVANT que vous le livriez. L'auto-revert annule le fine-tune s'il est pire que le modèle de base. Llama Guard scanne les sorties pour du contenu nuisible. Le niveau de risque EU AI Act que vous choisissez détermine quels artefacts de conformité sont auto-générés et si l'approbation humaine est REQUISE avant déploiement.", "wizard.step.operations.title": "Opérations et paramètres avancés", "wizard.step.operations.name": "Opérations", - "wizard.step.operations.desc": "Notifications, surveillance post-marché, génération de données synthétiques et horizons de rétention. Tous optionnels — passez ceux dont vous n'avez pas besoin.", - "wizard.step.operations.tutorial": "Ces quatre blocs couvrent tout ce que ForgeLM offre au-delà du flux core entraînement + compliance. Webhook envoie une alerte Slack/Teams/Discord au démarrage / succès / échec. La surveillance (Art. 72 / 17(1)(h)) enregistre un plan externe comme métadonnées ; ForgeLM n’exporte pas de métriques, ne détecte pas de dérive, n’envoie pas d’alertes de dérive et ne programme aucun contrôle. La génération synthétique laisse un modèle teacher plus fort produire des exemples supplémentaires. Les horizons de rétention fixent des TTL conformes RGPD pour audit log et artefacts éphémères. Aucun n'est obligatoire — laissez replié ce dont votre run n'a pas besoin.", + "wizard.step.operations.desc": "Notifications, métadonnées du plan de surveillance externe, données synthétiques et conservation. Configurez les options ForgeLM nécessaires.", + "wizard.step.operations.tutorial": "Ces quatre blocs couvrent tout ce que ForgeLM offre au-delà du flux core entraînement + compliance. Webhook envoie une alerte Slack/Teams/Discord au démarrage / succès / échec. La surveillance (Art. 72 / 17(1)(h)) enregistre un plan externe comme métadonnées ; ForgeLM n’exporte pas de métriques, ne détecte pas de dérive, n’envoie pas d’alertes de dérive et ne programme aucun contrôle. La génération synthétique laisse un modèle teacher plus fort produire des exemples supplémentaires. Les horizons de rétention fixent des TTL conformes RGPD pour audit log et artefacts éphémères. Ces paramètres ForgeLM sont facultatifs ; les obligations légales externes restent applicables.", "wizard.risk.label": "Classification de risque EU AI Act", "wizard.risk.hint": "Choisissez le niveau dans lequel votre déploiement tombe. High-risk et unacceptable déclenchent des obligations d'audit + approbation humaine obligatoires.", "wizard.risk.unknown": "Unknown — placeholder de pré-classification", @@ -2962,20 +2971,23 @@ "wizard.webhook.notify_success.label": "Notifier au succès", "wizard.webhook.notify_failure.label": "Notifier à l'échec", "wizard.monitoring.title": "Plan de surveillance externe (Art. 72 / 17(1)(h))", - "wizard.monitoring.hint": "Déclarez votre surveillance externe dans le manifeste de conformité / Annexe IV. Métadonnées uniquement ; ForgeLM ne surveille pas le modèle.", + "wizard.monitoring.hint": "Déclarez votre surveillance externe dans le manifeste de conformité / Annex IV. Métadonnées uniquement ; ForgeLM n’exécute aucune surveillance.", "wizard.monitoring.enabled.label": "Enregistrer les métadonnées du plan", - "wizard.monitoring.enabled.hint": "Facultatif pour chaque niveau de risque. Un plan enregistré ne prouve pas que la surveillance fonctionne.", - "wizard.monitoring.enabled.detail": "ForgeLM enregistre le plan avec declarative_only: true. Il ne contacte pas le point de terminaison, n’exporte pas de métriques, ne détecte pas de dérive, n’envoie pas d’alertes de dérive et ne programme aucun contrôle. Exploitez votre infrastructure externe séparément ; cette déclaration seule ne remplit pas une obligation de surveillance après commercialisation.", - "wizard.monitoring.endpoint.label": "Point de terminaison déclaré (facultatif)", - "wizard.monitoring.endpoint.hint": "Enregistré uniquement, jamais contacté. env:NAME devient $NAME sans lire la variable ; sans adresse, la valeur est null.", + "wizard.monitoring.enabled.hint": "L’enregistrement dans ForgeLM est facultatif. L’Art. 72 impose aux fournisseurs de systèmes d’IA à haut risque une surveillance après commercialisation.", + "wizard.monitoring.enabled.detail": "ForgeLM enregistre le plan avec declarative_only: true. Il ne contacte pas le endpoint, n’exporte pas de métriques, ne détecte pas de dérive, n’envoie pas d’alertes de dérive et ne programme aucun contrôle. Exploitez votre infrastructure externe séparément ; cette déclaration seule ne remplit pas une obligation de surveillance après commercialisation.", + "wizard.monitoring.endpoint.label": "Endpoint déclaré (facultatif)", + "wizard.monitoring.endpoint.hint": "Jamais contacté. Les preuves en texte clair ne gardent que le schéma et l’hôte des URL ; aucune information d’identification. Préférez env:NAME : seul $NAME est enregistré, jamais sa valeur.", "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com OU env:MONITORING_URL", - "wizard.monitoring.metrics_export.label": "Exportateur externe déclaré", + "wizard.monitoring.metrics_export.label": "Outil d’export externe déclaré", "wizard.monitoring.metrics_export.none": "Aucun", "wizard.monitoring.metrics_export.prometheus": "Prometheus", "wizard.monitoring.metrics_export.datadog": "Datadog", "wizard.monitoring.metrics_export.custom": "Webhook custom", "wizard.monitoring.alert_on_drift.label": "Déclarer les alertes de dérive externes", "wizard.monitoring.alert_on_drift.hint": "Enregistre la politique de votre infrastructure externe. ForgeLM ne détecte pas de dérive et n’envoie pas ces alertes.", + "wizard.error.env_ref.invalid": "Utilisez env:NAME avec un identifiant ASCII : lettre ou soulignement initial, puis lettres, chiffres ou soulignements.", + "wizard.monitoring.interval.label": "Intervalle des contrôles externes (heures, 1–720)", + "wizard.error.monitoring.interval.invalid": "Saisissez un nombre entier entre 1 et 720 heures.", "wizard.synthetic.title": "Génération données synthétiques (avancé)", "wizard.synthetic.hint": "Générez données entraînement teacher → student avec modèle plus fort.", "wizard.synthetic.enabled.label": "Générer données synthétiques", @@ -3253,8 +3265,8 @@ "wizard.step.compliance.tutorial": "ForgeLM verifica el modelo fine-tuned ANTES de que lo entregue. Auto-revert deshace el fine-tune si es peor que el modelo base. Llama Guard escanea las salidas en busca de contenido dañino. El nivel de riesgo EU AI Act que elija determina qué artefactos de cumplimiento se autogeneran y si la aprobación humana es REQUERIDA antes del despliegue.", "wizard.step.operations.title": "Operaciones y ajustes avanzados", "wizard.step.operations.name": "Operaciones", - "wizard.step.operations.desc": "Notificaciones, monitoreo post-mercado, generación de datos sintéticos y horizontes de retención. Todos opcionales — salte los que no necesite.", - "wizard.step.operations.tutorial": "Estos cuatro bloques cubren todo lo que ForgeLM ofrece más allá del flujo central de entrenamiento + compliance. Webhook envía alerta Slack/Teams/Discord en inicio / éxito / fallo. El monitoreo (Art. 72 / 17(1)(h)) registra un plan externo como metadatos; ForgeLM no exporta métricas, detecta deriva, envía alertas de deriva ni programa comprobaciones. La generación sintética permite a un modelo teacher más fuerte producir ejemplos adicionales. Los horizontes de retención fijan TTLs conformes a RGPD para audit log y artefactos efímeros. Ninguno es obligatorio — déjelos cerrados si su run no los necesita.", + "wizard.step.operations.desc": "Notificaciones, metadatos del plan de monitoreo externo, datos sintéticos y retención. Configure las opciones de ForgeLM que necesite.", + "wizard.step.operations.tutorial": "Estos cuatro bloques cubren todo lo que ForgeLM ofrece más allá del flujo central de entrenamiento + compliance. Webhook envía alerta Slack/Teams/Discord en inicio / éxito / fallo. El monitoreo (Art. 72 / 17(1)(h)) registra un plan externo como metadatos; ForgeLM no exporta métricas, no detecta deriva, no envía alertas de deriva ni programa comprobaciones. La generación sintética permite a un modelo teacher más fuerte producir ejemplos adicionales. Los horizontes de retención fijan TTLs conformes a RGPD para audit log y artefactos efímeros. Estas opciones de ForgeLM son opcionales; las obligaciones legales externas siguen vigentes.", "wizard.risk.label": "Clasificación de riesgo EU AI Act", "wizard.risk.hint": "Elija el nivel en el que cae su despliegue. High-risk y unacceptable activan obligaciones obligatorias de auditoría + aprobación humana.", "wizard.risk.unknown": "Unknown — placeholder de pre-clasificación", @@ -3356,12 +3368,12 @@ "wizard.webhook.notify_success.label": "Notificar en éxito", "wizard.webhook.notify_failure.label": "Notificar en fallo", "wizard.monitoring.title": "Plan de monitoreo externo (Art. 72 / 17(1)(h))", - "wizard.monitoring.hint": "Declare su monitoreo externo en el manifiesto de cumplimiento / Anexo IV. Solo metadatos; ForgeLM no ejecuta el monitoreo.", + "wizard.monitoring.hint": "Declare su monitoreo externo en el manifiesto de cumplimiento / Annex IV. Solo metadatos; ForgeLM no ejecuta el monitoreo.", "wizard.monitoring.enabled.label": "Registrar metadatos del plan", - "wizard.monitoring.enabled.hint": "Opcional para todos los niveles de riesgo. Registrar un plan no demuestra que el monitoreo esté funcionando.", - "wizard.monitoring.enabled.detail": "ForgeLM registra el plan con declarative_only: true. No contacta el endpoint, exporta métricas, detecta deriva, envía alertas de deriva ni programa comprobaciones. Opere su infraestructura externa por separado; esta declaración por sí sola no cumple una obligación de monitoreo posterior a la comercialización.", + "wizard.monitoring.enabled.hint": "El registro en ForgeLM es opcional. El Art. 72 exige a los proveedores de sistemas de IA de alto riesgo un sistema de monitoreo posterior a la comercialización.", + "wizard.monitoring.enabled.detail": "ForgeLM registra el plan con declarative_only: true. No contacta el endpoint, no exporta métricas, no detecta deriva, no envía alertas de deriva ni programa comprobaciones. Opere su infraestructura externa por separado; esta declaración por sí sola no cumple una obligación de monitoreo posterior a la comercialización.", "wizard.monitoring.endpoint.label": "Endpoint declarado (opcional)", - "wizard.monitoring.endpoint.hint": "Solo se registra, nunca se contacta. env:NAME se guarda como $NAME sin leer la variable; sin endpoint se registra null.", + "wizard.monitoring.endpoint.hint": "Nunca se contacta. La evidencia en texto claro conserva solo esquema/host de las URL; no incluya credenciales. Use env:NAME: solo se registra $NAME, nunca su valor.", "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com O env:MONITORING_URL", "wizard.monitoring.metrics_export.label": "Exportador externo declarado", "wizard.monitoring.metrics_export.none": "Ninguno", @@ -3370,6 +3382,9 @@ "wizard.monitoring.metrics_export.custom": "Webhook custom", "wizard.monitoring.alert_on_drift.label": "Declarar alertas externas de deriva", "wizard.monitoring.alert_on_drift.hint": "Registra la política de su infraestructura externa. ForgeLM no detecta deriva ni envía estas alertas.", + "wizard.error.env_ref.invalid": "Use env:NAME con un identificador ASCII: letra o guion bajo inicial, seguido de letras, dígitos o guiones bajos.", + "wizard.monitoring.interval.label": "Intervalo de comprobación externa (horas, 1–720)", + "wizard.error.monitoring.interval.invalid": "Introduzca un número entero de 1 a 720 horas.", "wizard.synthetic.title": "Generación datos sintéticos (avanzado)", "wizard.synthetic.hint": "Genere datos entrenamiento teacher → student usando modelo más fuerte.", "wizard.synthetic.enabled.label": "Generar datos sintéticos", @@ -3647,8 +3662,8 @@ "wizard.step.compliance.tutorial": "ForgeLM 在你交付之前检查微调后的模型。Auto-revert 在微调比基础模型差时撤销微调。Llama Guard 扫描输出中的有害内容。你选择的 EU AI Act 风险等级决定哪些合规 artefact 自动生成,以及在部署之前是否需要人工批准。", "wizard.step.operations.title": "运维与高级设置", "wizard.step.operations.name": "运维", - "wizard.step.operations.desc": "通知、上市后监控、合成数据生成和保留期限。全部可选 — 不需要的可以跳过。", - "wizard.step.operations.tutorial": "这四个区块涵盖了 ForgeLM 在核心训练 + 合规流程之外提供的一切。Webhook 在训练开始 / 成功 / 失败时发送 Slack/Teams/Discord 警报。监控（第 72 / 17(1)(h) 条）仅将外部计划记录为元数据；ForgeLM 不导出指标、检测漂移、发送漂移警报或安排检查。合成数据生成让更强的 teacher 模型产生额外训练样本。保留期限为审计日志和临时 artefact 设置符合 GDPR 的 TTL。没有任何项是强制的 — 你的训练不需要的话保持折叠状态即可。", + "wizard.step.operations.desc": "通知、外部监控计划元数据、合成数据和保留设置。配置所需的 ForgeLM 选项。", + "wizard.step.operations.tutorial": "这四个区块涵盖了 ForgeLM 在核心训练 + 合规流程之外提供的一切。Webhook 在训练开始 / 成功 / 失败时发送 Slack/Teams/Discord 警报。监控（第 72 条及第 17(1)(h) 条）仅将外部计划记录为元数据；ForgeLM 不导出指标、检测漂移、发送漂移警报或安排检查。合成数据生成让更强的 teacher 模型产生额外训练样本。保留期限为审计日志和临时 artefact 设置符合 GDPR 的 TTL。这些 ForgeLM 设置是可选的；外部法律义务仍然适用。", "wizard.risk.label": "EU AI Act 风险分类", "wizard.risk.hint": "选择你的部署所属的等级。High-risk 和 unacceptable 触发强制审计 + 人工批准义务。", "wizard.risk.unknown": "Unknown — 预分类占位符", @@ -3749,13 +3764,13 @@ "wizard.webhook.notify_start.label": "训练开始时通知", "wizard.webhook.notify_success.label": "成功时通知", "wizard.webhook.notify_failure.label": "失败时通知", - "wizard.monitoring.title": "外部监控计划（第 72 / 17(1)(h) 条）", + "wizard.monitoring.title": "外部监控计划（第 72 条及第 17(1)(h) 条）", "wizard.monitoring.hint": "在合规清单 / 附录 IV 中声明外部监控设置。仅记录元数据；ForgeLM 不运行监控。", "wizard.monitoring.enabled.label": "记录监控计划元数据", - "wizard.monitoring.enabled.hint": "所有风险级别均可选。记录计划并不能证明监控正在运行。", + "wizard.monitoring.enabled.hint": "在 ForgeLM 中记录元数据是可选的。第 72 条要求高风险 AI 系统提供者建立上市后监控系统。", "wizard.monitoring.enabled.detail": "ForgeLM 使用 declarative_only: true 标记记录计划。它不会连接端点、导出指标、检测漂移、发送漂移警报或安排检查。请单独运行外部监控系统；仅此声明并不能履行上市后监控义务。", "wizard.monitoring.endpoint.label": "声明的端点（可选）", - "wizard.monitoring.endpoint.hint": "仅记录，不连接。env:NAME 保存为 $NAME，不读取变量；未提供端点时记录 null。", + "wizard.monitoring.endpoint.hint": "不会连接。纯文本证据仅保留 URL 的协议和主机；请勿输入凭据。优先使用 env:NAME：只记录 $NAME，不读取变量值。", "wizard.monitoring.endpoint.placeholder": "https://monitor.example.com 或 env:MONITORING_URL", "wizard.monitoring.metrics_export.label": "声明的外部指标导出器", "wizard.monitoring.metrics_export.none": "无", @@ -3764,6 +3779,9 @@ "wizard.monitoring.metrics_export.custom": "自定义 webhook", "wizard.monitoring.alert_on_drift.label": "声明外部漂移警报策略", "wizard.monitoring.alert_on_drift.hint": "记录外部系统的策略。ForgeLM 不检测漂移，也不发送这些警报。", + "wizard.error.env_ref.invalid": "env:NAME 必须使用 ASCII 标识符：以字母或下划线开头，后接字母、数字或下划线。", + "wizard.monitoring.interval.label": "外部检查间隔（小时，1–720）", + "wizard.error.monitoring.interval.invalid": "请输入 1 到 720 之间的整数小时。", "wizard.synthetic.title": "合成数据生成(高级)", "wizard.synthetic.hint": "使用更强的模型生成 teacher → student 训练数据。", "wizard.synthetic.enabled.label": "生成合成数据", diff --git a/site/js/wizard.js b/site/js/wizard.js index 71c05d37..3098015f 100644 --- a/site/js/wizard.js +++ b/site/js/wizard.js @@ -256,6 +256,7 @@ monitoringEndpoint: '', metricsExport: 'none', // 'none' | 'prometheus' | 'datadog' | 'custom_webhook' alertOnDrift: true, + monitoringCheckIntervalHours: 24, // Synthetic data generation (teacher → student). Optional // pre-training stage; ``syntheticTeacherModel`` is the API // name or local path for the teacher. @@ -361,6 +362,14 @@ } } + // Shared ASCII env-reference syntax for monitoring metadata and webhook secrets. + function parseEnvRef(value) { + var raw = (value || '').trim(); + if (raw.slice(0, 4).toLowerCase() !== 'env:') return { literal: raw }; + var name = raw.slice(4).trim(); + return /^[A-Za-z_][A-Za-z0-9_]*$/.test(name) ? { name: name } : { invalid: true }; + } + /* ── Required-field validation ────────────────────────── */ // Some steps require operator input that has no sensible default — // e.g. a custom HuggingFace model ID, or a dataset path / Hub name. @@ -446,6 +455,15 @@ // Off → skip; the operator opted out of the whole feature. if (state.webhookEnabled && blank(state.webhookUrl)) { errors.push({ field: 'webhookUrl', key: 'wizard.error.webhook.required' }); + } else if (state.webhookEnabled && parseEnvRef(state.webhookUrl).invalid) { + errors.push({ field: 'webhookUrl', key: 'wizard.error.env_ref.invalid' }); + } + if (state.monitoringEnabled && parseEnvRef(state.monitoringEndpoint).invalid) { + errors.push({ field: 'monitoringEndpoint', key: 'wizard.error.env_ref.invalid' }); + } + if (state.monitoringEnabled && (!Number.isInteger(state.monitoringCheckIntervalHours) || + state.monitoringCheckIntervalHours < 1 || state.monitoringCheckIntervalHours > 720)) { + errors.push({ field: 'monitoringCheckIntervalHours', key: 'wizard.error.monitoring.interval.invalid' }); } if (state.syntheticEnabled) { if (blank(state.syntheticTeacherModel)) { @@ -802,11 +820,11 @@ // is a secret. if (state.webhookEnabled && state.webhookUrl) { header(0, 'webhook'); - var raw = state.webhookUrl.trim(); - if (raw.indexOf('env:') === 0) { - k(1, 'url_env', JSON.stringify(raw.slice(4))); - } else { - k(1, 'url', JSON.stringify(raw)); + var webhookRef = parseEnvRef(state.webhookUrl); + if (webhookRef.name) { + k(1, 'url_env', JSON.stringify(webhookRef.name)); + } else if (!webhookRef.invalid) { + k(1, 'url', JSON.stringify(webhookRef.literal)); } if (state.webhookNotifyStart) k(1, 'notify_on_start', 'true'); if (!state.webhookNotifySuccess) k(1, 'notify_on_success', 'false'); @@ -815,23 +833,26 @@ } // Record an external plan without activating monitoring hooks. - var monitoringActive = state.monitoringEnabled; - if (monitoringActive || state.monitoringEndpoint) { + if (state.monitoringEnabled) { comment('External monitoring plan metadata only; ForgeLM runs no exporters, drift alerts, or scheduled checks.'); header(0, 'monitoring'); - k(1, 'enabled', monitoringActive ? 'true' : 'false'); + k(1, 'enabled', 'true'); if (state.monitoringEndpoint) { - var rawM = state.monitoringEndpoint.trim(); - if (rawM.indexOf('env:') === 0) { - k(1, 'endpoint_env', JSON.stringify(rawM.slice(4))); - } else { - k(1, 'endpoint', JSON.stringify(rawM)); + var monitoringRef = parseEnvRef(state.monitoringEndpoint); + if (monitoringRef.name) { + k(1, 'endpoint_env', JSON.stringify(monitoringRef.name)); + } else if (!monitoringRef.invalid && monitoringRef.literal) { + k(1, 'endpoint', JSON.stringify(monitoringRef.literal)); } } if (state.metricsExport && state.metricsExport !== 'none') { k(1, 'metrics_export', JSON.stringify(state.metricsExport)); } - if (!state.alertOnDrift) k(1, 'alert_on_drift', 'false'); + k(1, 'alert_on_drift', state.alertOnDrift ? 'true' : 'false'); + if (Number.isInteger(state.monitoringCheckIntervalHours) && state.monitoringCheckIntervalHours >= 1 && + state.monitoringCheckIntervalHours <= 720) { + k(1, 'check_interval_hours', String(state.monitoringCheckIntervalHours)); + } blank(); } @@ -1116,6 +1137,7 @@ judgeApiKeyEnv: 'eval_gates', webhookUrl: 'webhook', monitoringEndpoint: 'monitoring', + monitoringCheckIntervalHours: 'monitoring', syntheticTeacherModel: 'synthetic', syntheticSeedFile: 'synthetic', syntheticApiKeyEnv: 'synthetic' @@ -2717,9 +2739,10 @@ var endpointShown = !!state.monitoringEnabled; if (endpointShown) { + var monEndpointErr = fieldHasError(state, 'operations', 'monitoringEndpoint'); var epInput = el('input', { type: 'text', - class: 'wizard-input', + class: 'wizard-input' + (monEndpointErr ? ' is-invalid' : ''), placeholder: tr('wizard.monitoring.endpoint.placeholder'), value: state.monitoringEndpoint || '' }); @@ -2729,6 +2752,7 @@ el('span', { class: 'wizard-row-hint', text: tr('wizard.monitoring.endpoint.hint') }) ]; var epRow = el('div', { class: 'wizard-row' }, epRowChildren); + if (monEndpointErr) epRow.appendChild(el('div', { class: 'wizard-error', role: 'alert', text: tr(monEndpointErr) })); epInput.addEventListener('input', function () { state.monitoringEndpoint = epInput.value.trim(); persist(); @@ -2736,6 +2760,24 @@ }); body.appendChild(epRow); + var intervalError = fieldHasError(state, 'operations', 'monitoringCheckIntervalHours'); + var intervalInput = el('input', { + id: 'wizard-monitoring-interval', type: 'number', min: '1', max: '720', step: '1', + class: 'wizard-input' + (intervalError ? ' is-invalid' : ''), + value: String(state.monitoringCheckIntervalHours) + }); + intervalInput.addEventListener('input', function () { + state.monitoringCheckIntervalHours = Number(intervalInput.value); + persist(); + liveYaml(); + }); + var intervalRow = el('div', { class: 'wizard-row' }, [ + el('label', { for: 'wizard-monitoring-interval', class: 'wizard-row-label', text: tr('wizard.monitoring.interval.label') }), + intervalInput + ]); + if (intervalError) intervalRow.appendChild(el('div', { class: 'wizard-error', role: 'alert', text: tr(intervalError) })); + body.appendChild(intervalRow); + var meSelect = el('select', { class: 'wizard-select' }); [ { id: 'none', labelKey: 'wizard.monitoring.metrics_export.none' }, diff --git a/tests/site/wizard_monitoring.test.cjs b/tests/site/wizard_monitoring.test.cjs new file mode 100644 index 00000000..fcfcdf8f --- /dev/null +++ b/tests/site/wizard_monitoring.test.cjs @@ -0,0 +1,145 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const { spawnSync } = require('node:child_process'); +const { test } = require('node:test'); +const vm = require('node:vm'); + +const root = path.resolve(__dirname, '../..'); +function element(tag) { + return { + tag, children: [], attrs: {}, events: {}, + appendChild(child) { this.children.push(child); }, + setAttribute(key, value) { this.attrs[key] = value; }, + addEventListener(key, handler) { this.events[key] = handler; }, + classList: { add() {}, toggle() {} }, + querySelector() { return null; } + }; +} +function loadWizard() { + const document = { + readyState: 'loading', addEventListener() {}, querySelector() { return null; }, createElement: element, + createElementNS: (_, tag) => element(tag), createTextNode: text => ({ text }) + }; + const context = vm.createContext({ window: {}, document }); + for (const file of ['wizard_defaults.js', 'translations.js']) { + vm.runInContext(fs.readFileSync(path.join(root, 'site/js', file), 'utf8'), context); + } + const marker = ' /* ── Boot ─'; + const source = fs.readFileSync(path.join(root, 'site/js/wizard.js'), 'utf8'); + assert.equal(source.split(marker).length, 2, 'test seam must identify the boot boundary exactly once'); + // Expose closure functions only in the VM; production has no test-only API. + vm.runInContext(source.replace(marker, + ' window.testWizard = {defaultState, buildYaml, validateStep, coerceForRisk, parseEnvRef, STEP_RENDERERS};\n' + marker), context); + return { ...context.window.testWizard, translations: context.window.ForgeLMTranslations }; +} +function state(wizard, overrides = {}) { + return Object.assign(wizard.defaultState(), { model: 'org/model', datasetName: 'org/dataset' }, overrides); +} +function descendants(node) { + return [node, ...(node.children || []).flatMap(descendants)]; +} + +for (const tier of ['minimal-risk', 'limited-risk', 'high-risk', 'unacceptable']) { + test(`monitoring remains opt-in at ${tier}; empty endpoint is valid`, () => { + const wizard = loadWizard(); + const value = state(wizard, { riskClassification: tier }); + wizard.coerceForRisk(value); + assert.equal(value.monitoringEnabled, false); + assert.doesNotMatch(wizard.buildYaml(value), /\nmonitoring:/); + value.monitoringEnabled = true; + assert.equal(wizard.validateStep(value, 'operations').length, 0); + assert.match(wizard.buildYaml(value), /monitoring:\n enabled: true/); + assert.doesNotMatch(wizard.buildYaml(value), /\n endpoint(?:_env)?:/); + }); +} + +for (const [raw, expected] of [['env:MON_URL', 'MON_URL'], ['ENV: MON_URL ', 'MON_URL'], [' env:_mon_url ', '_mon_url']]) { + test(`both endpoint inputs normalize ${JSON.stringify(raw)}`, () => { + const wizard = loadWizard(); + const value = state(wizard, { monitoringEnabled: true, monitoringEndpoint: raw, webhookEnabled: true, webhookUrl: raw }); + assert.equal(wizard.validateStep(value, 'operations').length, 0); + const yaml = wizard.buildYaml(value); + assert.ok(yaml.includes(`endpoint_env: "${expected}"`)); + assert.ok(yaml.includes(`url_env: "${expected}"`)); + }); +} +for (const raw of ['env:', 'env: ', 'env:$MON_URL', 'env:https://host/token', 'env:MON URL', 'env:1MON', 'env:MÖN']) { + test(`both endpoint inputs reject ${JSON.stringify(raw)}`, () => { + const wizard = loadWizard(); + const value = state(wizard, { monitoringEnabled: true, monitoringEndpoint: raw, webhookEnabled: true, webhookUrl: raw }); + const fields = Array.from(wizard.validateStep(value, 'operations'), item => item.field); + assert.deepEqual(fields.sort(), ['monitoringEndpoint', 'webhookUrl']); + assert.doesNotMatch(wizard.buildYaml(value), /(?:endpoint_env|url_env):/); + }); +} + +test('disabled plans omit stale endpoint, exporter, and drift policy', () => { + const wizard = loadWizard(); + const value = state(wizard, { monitoringEnabled: false, monitoringEndpoint: 'env:MON_URL', metricsExport: 'prometheus', alertOnDrift: true }); + assert.doesNotMatch(wizard.buildYaml(value), /\nmonitoring:/); +}); + +for (const interval of [0, 721, 1.5, NaN]) { + test(`invalid external check interval ${interval} blocks advancement`, () => { + const wizard = loadWizard(); + const value = state(wizard, { monitoringEnabled: true, monitoringCheckIntervalHours: interval }); + assert.equal(wizard.validateStep(value, 'operations')[0].field, 'monitoringCheckIntervalHours'); + assert.doesNotMatch(wizard.buildYaml(value), /check_interval_hours:/); + }); +} + +test('operations renderer accepts plan edits and reports invalid env references', () => { + const wizard = loadWizard(); + const value = state(wizard, { monitoringEnabled: true, monitoringEndpoint: 'env:', attemptedSteps: ['operations'] }); + const pane = element('main'); + wizard.STEP_RENDERERS.operations(pane, value, () => {}, () => {}); + const nodes = descendants(pane); + assert.ok(nodes.some(node => node.attrs.role === 'alert' && node.textContent === 'wizard.error.env_ref.invalid')); + const interval = nodes.find(node => node.attrs.id === 'wizard-monitoring-interval'); + interval.value = '48'; + interval.events.input(); + assert.equal(value.monitoringCheckIntervalHours, 48); + assert.match(wizard.buildYaml(value), /check_interval_hours: 48/); + const toggle = nodes.find(node => node.tag === 'label' && node.children.some(child => + child.textContent === 'wizard.monitoring.enabled.label')).children[0]; + toggle.checked = false; + toggle.events.change(); + assert.doesNotMatch(wizard.buildYaml(value), /\nmonitoring:/); +}); + +test('all six locales expose the same monitoring and validation keys', () => { + const { translations } = loadWizard(); + const keys = Object.keys(translations.en).filter(key => key.startsWith('wizard.monitoring.') || + key === 'wizard.error.env_ref.invalid' || key === 'wizard.error.monitoring.interval.invalid').sort(); + for (const locale of ['en', 'tr', 'de', 'fr', 'es', 'zh']) { + assert.deepEqual(Object.keys(translations[locale]).filter(key => keys.includes(key)).sort(), keys); + assert.ok(translations[locale]['wizard.monitoring.enabled.detail'].includes('declarative_only: true')); + for (const key of keys) assert.ok(translations[locale][key].trim(), `${locale}: ${key}`); + } +}); + +test('generated monitoring YAML passes the real ForgeConfig schema', () => { + const wizard = loadWizard(); + const configs = []; + for (const tier of ['minimal-risk', 'limited-risk', 'high-risk', 'unacceptable']) { + for (const endpoint of ['', 'ENV: MON_URL ', 'https://monitor.example.com/plan']) { + const value = state(wizard, { riskClassification: tier, monitoringEnabled: true, monitoringEndpoint: endpoint, monitoringCheckIntervalHours: 720 }); + wizard.coerceForRisk(value); + configs.push(wizard.buildYaml(value)); + } + } + const result = spawnSync('python3', ['-c', [ + 'import json, sys, yaml', + 'from forgelm.config import ForgeConfig', + 'for text in json.load(sys.stdin):', + ' config = ForgeConfig(**yaml.safe_load(text))', + ' assert config.monitoring.enabled is True', + ' assert config.monitoring.check_interval_hours == 720', + ' assert {"alert_on_drift", "check_interval_hours"} <= config.monitoring.model_fields_set' + ].join('\n')], { cwd: root, input: JSON.stringify(configs), encoding: 'utf8', timeout: 30000 }); + assert.ifError(result.error); + assert.equal(result.status, 0, result.stderr); +}); diff --git a/tests/test_config.py b/tests/test_config.py index 903390f5..e4caac1d 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -27,11 +27,13 @@ MergeConfig, ModelConfig, MonitoringConfig, + PipelineStage, SafetyConfig, SyntheticConfig, TrainingConfig, WebhookConfig, load_config, + merge_pipeline_stage_config, ) from tests._helpers.factories import minimal_config @@ -83,7 +85,12 @@ class TestMonitoringConfig: @pytest.mark.parametrize("exporter", ["prometheus", "datadog", "custom_webhook"]) def test_exporter_declaration_warns_at_config_load(self, tmp_path, caplog, exporter): config_data = minimal_config( - monitoring={"enabled": True, "metrics_export": exporter, "endpoint": "https://secret.example.com/private"} + monitoring={ + "enabled": True, + "metrics_export": exporter, + "endpoint": "https://secret.example.com/private", + "endpoint_env": "PRIVATE_MONITORING_NAME", + } ) path = _write_yaml(config_data, str(tmp_path / "monitoring.yaml")) with caplog.at_level(logging.WARNING, logger="forgelm.config"): @@ -93,6 +100,29 @@ def test_exporter_declaration_warns_at_config_load(self, tmp_path, caplog, expor assert "declarative metadata only" in caplog.text assert "does not export metrics" in caplog.text assert "secret.example.com" not in caplog.text + assert "PRIVATE_MONITORING_NAME" not in caplog.text + assert sum("monitoring.metrics_export" in record.message for record in caplog.records) == 1 + for name in ("first", "second", "third"): + merge_pipeline_stage_config(config, PipelineStage(name=name)) + assert sum("monitoring.metrics_export" in record.message for record in caplog.records) == 1 + + @pytest.mark.parametrize("value", ["MON_URL", " _mon_url ", "A1"]) + def test_env_reference_is_normalized_without_resolution(self, value, monkeypatch): + monkeypatch.setenv(value.strip(), "https://secret.example.com/token") + assert MonitoringConfig(endpoint_env=value).endpoint_env == value.strip() + + @pytest.mark.parametrize("value", [None, "", " ", "\t\n"]) + def test_blank_env_reference_is_absent(self, value): + assert MonitoringConfig(endpoint_env=value).endpoint_env is None + + @pytest.mark.parametrize("value", ["env:MON_URL", "$MON_URL", "https://host/token", "MON URL", "1MON", "MÖN"]) + def test_invalid_env_reference_is_rejected(self, value): + with pytest.raises(ValidationError, match="ASCII environment-variable name"): + MonitoringConfig(endpoint_env=value) + + @pytest.mark.parametrize("value,expected", [(" ", ""), (" https://host/path\n", "https://host/path")]) + def test_literal_endpoint_is_trimmed(self, value, expected): + assert MonitoringConfig(endpoint=value).endpoint == expected @pytest.mark.parametrize( "monitoring", diff --git a/tests/test_eu_ai_act.py b/tests/test_eu_ai_act.py index 9a102a06..6e83758b 100644 --- a/tests/test_eu_ai_act.py +++ b/tests/test_eu_ai_act.py @@ -25,6 +25,7 @@ RiskAssessmentConfig, load_config, ) +from forgelm.verify import verify_annex_iv_artifact # --- Config Models --- @@ -816,16 +817,23 @@ def test_successful_export_promotes_all_artifacts(self, tmp_path): class TestManifestAnnexIV: - @pytest.mark.parametrize("env_value", [None, "https://secret.example.com/private"]) + @pytest.mark.parametrize("env_value", [None, "https://secret.example.com/private"], ids=["env-unset", "env-set"]) @pytest.mark.parametrize( "endpoint_fields,expected", [ ({}, None), ({"endpoint": "https://monitor.example.com"}, "https://monitor.example.com"), + ( + {"endpoint": " https://user:password@monitor.example.com/secret?token=credential#fragment "}, + "https://monitor.example.com", + ), ({"endpoint_env": "MON_URL"}, "$MON_URL"), ({"endpoint": "https://monitor.example.com", "endpoint_env": "MON_URL"}, "$MON_URL"), ({"endpoint_env": ""}, None), + ({"endpoint_env": " ", "endpoint": " "}, None), + ({"endpoint_env": " MON_URL "}, "$MON_URL"), ], + ids=["absent", "literal", "masked-literal", "env", "env-precedence", "blank-env", "whitespace", "trimmed-env"], ) def test_monitoring_declaration_survives_export( self, tmp_path, minimal_config, monkeypatch, endpoint_fields, expected, env_value @@ -851,9 +859,11 @@ def test_monitoring_declaration_survives_export( declaration = { "declarative_only": True, "endpoint": expected, + "endpoint_redacted": expected == "https://monitor.example.com", "metrics_export": "prometheus", "alert_on_drift": True, "check_interval_hours": 12, + "defaulted_fields": ["endpoint"] if "endpoint" not in endpoint_fields and expected is None else [], } assert manifest["monitoring"] == declaration export_compliance_artifacts(manifest, str(tmp_path)) @@ -863,7 +873,40 @@ def test_monitoring_declaration_survives_export( assert annex["monitoring_and_logging"] == declaration assert "secret.example.com" not in json.dumps(report) assert "secret.example.com" not in json.dumps(annex) + for secret in ("password", "credential", "/secret", "#fragment"): + assert secret not in json.dumps(report) + assert secret not in json.dumps(annex) + assert secret not in (tmp_path / "training_manifest.yaml").read_text(encoding="utf-8") assert annex["metadata"]["manifest_hash"] == compute_annex_iv_manifest_hash(annex) + assert verify_annex_iv_artifact(str(tmp_path / "annex_iv_metadata.json")).valid + + def test_monitoring_defaults_are_not_operator_declarations(self, minimal_config): + from forgelm.config import PipelineStage, merge_pipeline_stage_config + + config = ForgeConfig(**minimal_config(monitoring={"enabled": True})) + for effective in (config, merge_pipeline_stage_config(config, PipelineStage(name="first"))): + declaration = generate_training_manifest(effective, {})["monitoring"] + assert declaration["defaulted_fields"] == [ + "endpoint", + "metrics_export", + "alert_on_drift", + "check_interval_hours", + ] + assert declaration["alert_on_drift"] is True + assert declaration["check_interval_hours"] == 24 + + explicit = ForgeConfig( + **minimal_config( + monitoring={ + "enabled": True, + "endpoint": "", + "metrics_export": "none", + "alert_on_drift": True, + "check_interval_hours": 24, + } + ) + ) + assert generate_training_manifest(explicit, {})["monitoring"]["defaulted_fields"] == [] @pytest.mark.parametrize("monitoring", [None, {"enabled": False, "metrics_export": "datadog"}]) def test_disabled_monitoring_is_not_declared(self, minimal_config, monitoring): diff --git a/tests/test_guard_wiring.py b/tests/test_guard_wiring.py index 90392f3a..b96fa088 100644 --- a/tests/test_guard_wiring.py +++ b/tests/test_guard_wiring.py @@ -354,6 +354,7 @@ def test_gauntlet_non_guard_steps_are_wired_into_ci(): "ruff check .", "ruff format", "pytest", + "node --test tests/site/*.test.cjs", "--config config_template.yaml --dry-run", # The FULL mypy argv, including the probe. A fragment stopping one # argument short was the bug: deleting ` tests/typing/public_surface_probe.py` diff --git a/tests/test_wizard_phase22.py b/tests/test_wizard_phase22.py index 107afc24..850f995e 100644 --- a/tests/test_wizard_phase22.py +++ b/tests/test_wizard_phase22.py @@ -78,11 +78,8 @@ def test_env_prefix_empty_var_name_rejected(self): with pytest.raises(ValueError, match="non-empty variable name"): wizard._parse_webhook_value("env:") - def test_env_prefix_lowercase_var_name_rejected(self): - # POSIX env-var convention enforces uppercase letters; lower- - # case names break ``${VAR}`` shell expansion in CI. - with pytest.raises(ValueError, match="POSIX environment-variable name"): - wizard._parse_webhook_value("env:lowercase_var") + def test_env_prefix_lowercase_var_name_accepted(self): + assert wizard._parse_webhook_value("env:lowercase_var") == {"url_env": "lowercase_var"} def test_bare_string_no_scheme_rejected(self): with pytest.raises(ValueError, match="not a valid URL"): @@ -1011,11 +1008,13 @@ def test_empty_var_name_raises(self): with pytest.raises(ValueError, match="non-empty variable name"): _parse_env_ref("env:") - def test_lowercase_var_name_raises(self): + @pytest.mark.parametrize( + "reference,expected", [("env:lowercase_var", "lowercase_var"), ("ENV: _MON_URL ", "_MON_URL")] + ) + def test_ascii_identifiers_are_accepted(self, reference, expected): from forgelm.wizard._collectors import _parse_env_ref - with pytest.raises(ValueError, match="POSIX environment-variable name"): - _parse_env_ref("env:lowercase_var") + assert _parse_env_ref(reference) == expected def test_monitoring_collector_reuses_shared_helper(self, capsys): # _collect_monitoring's soft-fail policy (warn + drop the field, @@ -1199,7 +1198,7 @@ def test_metadata_plan_accepts_no_endpoint(self, capsys): assert "does not export metrics" in output assert "schedule checks" in output - def test_env_prefix_routes_to_endpoint_env(self): + def test_env_prefix_routes_to_endpoint_env(self, capsys): with patch( "builtins.input", side_effect=_input_returning( @@ -1213,6 +1212,7 @@ def test_env_prefix_routes_to_endpoint_env(self): result = wizard._collect_monitoring() assert result["endpoint_env"] == "DATADOG_URL" assert "endpoint" not in result + assert "Metadata only" in capsys.readouterr().out def test_literal_url_routes_to_endpoint(self): with patch( diff --git a/tools/check_module_size.py b/tools/check_module_size.py index 4feb73da..f3b89dfe 100644 --- a/tools/check_module_size.py +++ b/tools/check_module_size.py @@ -263,7 +263,7 @@ class _DeferredSplit: # _validate_entries() sees a raise without reading the diff. _DEFERRED_SPLITS: dict[str, _DeferredSplit] = { "forgelm/compliance.py": _DeferredSplit( - budget=2493, + budget=2505, deferred_at_loc=2147, reason=( "EU AI Act Art. 9-17 + Annex IV builder + hash-chained audit log + " @@ -271,6 +271,7 @@ class _DeferredSplit: "_annex_iv, _provenance, _gdpr." ), budget_history=( + "2026-10-15 (#93 review, 2493 -> 2505, +12): Persist monitoring default provenance and masked endpoint metadata at the existing writer; an unrelated compliance split would obscure this evidence-contract correction.", "2026-10-15 (#93, 2492 -> 2493, +1): label the persisted monitoring plan " "declarative_only so neither the manifest nor its Annex IV projection implies " "runtime monitoring. Keep the label with the existing writer; the one-line " @@ -317,9 +318,10 @@ class _DeferredSplit: ), ), "forgelm/config.py": _DeferredSplit( - budget=1870, + budget=1887, deferred_at_loc=1795, budget_history=( + "2026-10-15 (#93 review, 1870 -> 1887, +17): Normalize and validate monitoring endpoints beside their schema and emit the exporter warning at load_config, avoiding pipeline revalidation duplicates without mutable state.", "2026-10-15 (#93, 1855 -> 1870, +15): describe MonitoringConfig as external-plan " "metadata and warn when an enabled plan declares an exporter ForgeLM does not run. " "The validator belongs beside its schema; the remaining growth is required field " From b03fea18876ca3a85a7787551d1ce4c3013c8ba1 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 15 Oct 2026 22:05:13 +0300 Subject: [PATCH 03/44] docs: close monitoring and webhook review contradictions Identify monitoring as Annex IV point 9 / Article 72(3), and risk management as point 5, in the touched EN/TR reference tables. Distinguish metadata summaries from the full external monitoring plan and remove stale wizard Article 12+17 labels from current user-facing references. Replace nonexistent webhook_failed audit events with forgelm.webhook WARNING diagnostics, correct allow_private_destinations, and qualify lifecycle notifications and evaluation trends in compliance summaries. The remaining Annex IV mapping is tracked by #539; broader verifier completeness and audit-reference behavior remain in #573 and #658. Resolves DOC-D3B04B-013 and remaining monitoring documentation surfaces of DOC-D1B01-004, DOC-D3B04A-002, DOC-D4B02-003, DOC-D4B09-008. Fixes #231 Refs #93, #539, #946 Co-authored-by: Codex --- docs/design/iso27001_soc2_alignment.md | 2 +- docs/guides/quickstart-tr.md | 2 +- docs/guides/quickstart.md | 2 +- docs/guides/safety_compliance-tr.md | 2 +- docs/guides/safety_compliance.md | 2 +- docs/reference/architecture-tr.md | 2 +- docs/reference/architecture.md | 2 +- docs/reference/compliance_summary-tr.md | 2 +- docs/reference/compliance_summary.md | 2 +- docs/reference/verify_annex_iv_subcommand-tr.md | 4 ++-- docs/reference/verify_annex_iv_subcommand.md | 4 ++-- docs/usermanuals/en/compliance/annex-iv.md | 4 ++-- docs/usermanuals/en/compliance/overview.md | 6 +++--- docs/usermanuals/en/operations/air-gap.md | 2 +- docs/usermanuals/en/operations/troubleshooting.md | 6 +++--- docs/usermanuals/tr/compliance/annex-iv.md | 4 ++-- docs/usermanuals/tr/compliance/overview.md | 6 +++--- docs/usermanuals/tr/operations/air-gap.md | 2 +- docs/usermanuals/tr/operations/troubleshooting.md | 6 +++--- 19 files changed, 31 insertions(+), 31 deletions(-) diff --git a/docs/design/iso27001_soc2_alignment.md b/docs/design/iso27001_soc2_alignment.md index 49483942..52805513 100644 --- a/docs/design/iso27001_soc2_alignment.md +++ b/docs/design/iso27001_soc2_alignment.md @@ -244,7 +244,7 @@ ForgeLM contributes nothing here. | A.8.13 Information backup | FL-helps | `audit_log.jsonl` + `.manifest.json` form a backupable unit; Annex IV bundle (`forgelm --compliance-export`) | Off-site backup with encryption | | A.8.14 Redundancy of information processing facilities | OOS | — | Multi-AZ infra | | A.8.15 Logging | FL | `audit_log.jsonl` JSON Lines + HMAC chain + genesis manifest; `forgelm verify-audit` validates | Ship to SIEM | -| A.8.16 Monitoring activities | FL-helps | Webhook lifecycle events; `safety_trend.jsonl` cross-run trend tracking | Define alert thresholds | +| A.8.16 Monitoring activities | FL-helps | Webhook lifecycle notifications and `safety_trend.jsonl` evaluation trends; external monitoring-plan metadata only | Operate monitoring and define alert thresholds | | A.8.17 Clock synchronisation | FL-helps | All audit entries ISO-8601 UTC | NTP on training hosts | | A.8.18 Use of privileged utility programs | OOS | — | OS-level privilege management | | A.8.19 Installation of software on operational systems | FL-helps | `forgelm doctor` reports installed packages + versions; `pyproject.toml` pins; `pip install forgelm==X.Y.Z` | Package allowlist | diff --git a/docs/guides/quickstart-tr.md b/docs/guides/quickstart-tr.md index ed185a57..43b6ab2a 100644 --- a/docs/guides/quickstart-tr.md +++ b/docs/guides/quickstart-tr.md @@ -72,7 +72,7 @@ Bundled seed dataset'lerin lisansları için bkz. [LICENSES.md](https://github.c forgelm --wizard ``` -Sihirbaz önce curated quickstart-template kısayolu önerir; reddedilirse 9 adımlı etkileşimli akış açılır (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations) ve her `ForgeConfig` bloğunu kapsar — model, LoRA / DoRA / PiSSA / rsLoRA / GaLore stratejisi, trainer-spesifik hyperparam'lar (`dpo_beta` / `simpo_*` / `kto_beta` / `orpo_beta` / `grpo_*`), EU AI Act Madde 9 / 10 / 11 / 12+17 uyumluluk metadata, retention, monitoring, evaluation kapıları, webhook, sentetik veri — ve kullanıma-hazır bir YAML yazar. Geri dönmek için `back` / `b`, sıfırlamak için `reset` / `r`; state `~/.cache/forgelm/wizard_state.yaml`'da persistent, Ctrl-C / yeni oturum kaldığı yerden devam edebilir. +Sihirbaz önce curated quickstart-template kısayolu önerir; reddedilirse 9 adımlı etkileşimli akış açılır (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations) ve her `ForgeConfig` bloğunu kapsar — model, LoRA / DoRA / PiSSA / rsLoRA / GaLore stratejisi, trainer-spesifik hyperparam'lar (`dpo_beta` / `simpo_*` / `kto_beta` / `orpo_beta` / `grpo_*`), EU AI Act Madde 9 / 10 / 11 / 12 / 17(1)(h) / 72 uyumluluk metadata, retention, monitoring, evaluation kapıları, webhook, sentetik veri — ve kullanıma-hazır bir YAML yazar. Geri dönmek için `back` / `b`, sıfırlamak için `reset` / `r`; state `~/.cache/forgelm/wizard_state.yaml`'da persistent, Ctrl-C / yeni oturum kaldığı yerden devam edebilir. Review-cycle 2 (2026-05-09) ile eklenen operatör guardrail'ları: sihirbaz çıkmadan önce kayıtlı YAML'ı `ForgeConfig.model_validate` ile doğrular (schema reddetmeleri 30 dakika sonra eğitim sırasında değil, anında yüzeylenir), mevcut config dosyasını üzerine yazmadan önce sorar (reddederseniz `_2.yaml` / `_3.yaml` ile auto-suffix'ler), non-tty stdin altında çalışmayı reddeder (scriptli koşumlar için `forgelm quickstart