MonitoringConfig is inert: no exporter, drift alert or schedule exists, yet the manifest and Annex IV §6 record it as configured (and '$None' when no endpoint is set) (CR-A01-007) #93

Wave 0 · theme D0-5: Documented controls that are not implemented · highest severity Critical · 12 finding(s) · effort large

Summary

An operator enables monitoring with metrics_export: prometheus and alert_on_drift: true. The Annex IV bundle states that post-market monitoring is configured, but no hook, export or alert ever runs.

Primary finding

CR-A01-007 — MonitoringConfig is inert: no exporter, drift alert or schedule exists, yet the manifest and Annex IV §6 record it as configured (and '$None' when no endpoint is set)

severity High · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. MonitoringConfig (forgelm/config.py:1279-1298) describes post-market monitoring hooks, Prometheus/Datadog/custom exporters, drift alerts and a check interval. Its only runtime consumer, generate_training_manifest (compliance.py:1581-1589), copies the fields into manifest['monitoring']. That becomes the Annex IV monitoring_and_logging section (compliance.py:1857). Nothing posts, exports, detects drift or schedules checks. mon_cfg.endpoint or f"${mon_cfg.endpoint_env}" records '$None' when neither field is set, and lets endpoint win over endpoint_env, contrary to the field description. The QMS SOP and the docs cite drift alerts that cannot fire.

Where.

• forgelm/config.py:1287-1289 (endpoint_env 'overrides endpoint' description contradicted by compliance.py:1585)
• forgelm/compliance.py:1581-1588 (the '$None' expression is on line 1585)
• forgelm/compliance.py:1857 (Annex IV monitoring_and_logging mapping; the comment is on 1856)
• forgelm/wizard/_collectors.py:894-912 (prompts and endpoint handling)
• docs/reference/configuration.md:359-369
• config_template.yaml:258-264
• forgelm/compliance.py:1582-1589
• forgelm/compliance.py:1857
• forgelm/compliance.py:1581-1589 (confirmed)
• docs/usermanuals/en/reference/configuration.md:277-287

Evidence.

• rg for monitoring/metrics_export/alert_on_drift/check_interval_hours outside config.py finds only compliance.py:1582-1588, the Annex IV section mapping, model_card.py:296 (exclusion) and the wizard. Dynamic: generate_training_manifest(cfg with monitoring.enabled only) returned endpoint '$None'. With endpoint='https://a' and endpoint_env='MON_URL' it returned 'https://a'.
• run1 (monitoring{enabled:true} only): compliance_report.json monitoring = {'endpoint': '$None', 'metrics_export': 'none', 'alert_on_drift': True, 'check_interval_hours': 24}; annex_iv_metadata.json:76-81 identical.
• rg for monitoring consumers in forgelm/ finds only compliance.py:1582-1589 (manifest copy), compliance.py:1857 (Annex IV section), model_card.py:296 (exclusion) and the wizard collector. There is no safe_post or other HTTP call that uses mon_cfg.

Recommended fix. Mark the block as declarative metadata in its description and docs (no runtime hooks), fix the '$None' rendering and the endpoint/endpoint_env precedence, or implement the hooks.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T09-003 (technology, Now) — Experiment tracking, webhooks and monitoring: monitoring.metrics_export prometheus / datadog / custom_webhook with no exporter (forgelm/config.py:1291) (incorrect-claim)
  severity Now
  Item. monitoring.metrics_export prometheus / datadog / custom_webhook with no exporter (forgelm/config.py:1291) · verdict: incorrect-claim · urgency: Now
  State in ForgeLM. forgelm/config.py:1291-1294 accepts none/prometheus/datadog/custom_webhook. Outside the wizard (forgelm/wizard/_collectors.py:894-903, _orchestrator.py:646), the only consumer is forgelm/compliance.py:1582-1589. It copies endpoint, metrics_export, alert_on_drift and check_interval_hours into the compliance manifest, and compliance.py:1857 puts them into the Annex IV §6 'monitoring_and_logging' fi…
  State of the ecosystem on 2026-10-01. Prometheus pulls by scraping. Its docs say 'Usually, the only valid use case for the Pushgateway is for capturing the outcome of a service-level batch job'. The Pushgateway takes PUT/POST to /metrics/job/ in the text (or protobuf) exposition format, not arbitrary JSON. Prometheus also has an opt-in OTLP receiver (--web.enable-otlp-receiver, /api/v1/otlp/v1/metrics). Datadog takes metrics thr…
  Recommendation. Now: rewrite the Field descriptions, config_template comment, both configuration references and the site/wizard copy in all languages to say these fields are declarative metadata recorded in the compliance manifest, and that ForgeLM exports nothing and has no drift detector. Add a config-load warning when monitoring.enabled is true and metrics_export is not none. Next: either implement a real exp…
  Sources.

  • Prometheus: When to use the Pushgateway (retrieved 2026-10-01 (page undated))
  • Prometheus Pushgateway README (/metrics/job/<JOB_NAME> push API) (retrieved 2026-10-01 (master))

• DOC-D1B01-004 (docs, Critical) — docs/reference/configuration.md: false claim — monitoring (EU AI Act Art. 12+17): enabled "Enable monitoring hooks", endpoint "Monitoring webhook URL", metrics_export prometheus/datadog/custo…
  severity Critical
  Where. docs/reference/configuration.md:360-369 (Turkish mirror: docs/reference/configuration-tr.md:302-311) · type: false claim
  The document says. monitoring (EU AI Act Art. 12+17): enabled "Enable monitoring hooks", endpoint "Monitoring webhook URL", metrics_export prometheus/datadog/custom_webhook, alert_on_drift "Alert on model drift", check_interval_hours "Monitoring check interval"
  What the code does. The block is metadata only. When enabled it is copied into the compliance manifest's 'monitoring' section (compliance.py:1582-1589). No code contacts the endpoint, exports metrics, detects drift, sends alerts or schedules checks; apart from the wizard prompts there is no other consumer in forgelm/.
  Code references. forgelm/compliance.py:1582-1589, forgelm/config.py:1279-1300
  Proposed correction. Fix the code (implement the hooks) or make the schema explicitly declarative. Temporary doc text: "The monitoring block is declarative metadata: when enabled: true it is copied into the compliance manifest so the Annex IV bundle documents your post-market monitoring plan. ForgeLM itself does not contact the endpoint, export metrics, detect dri…
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B04A-002 (docs, Critical) — docs/usermanuals/en/reference/configuration.md: false claim — monitoring: — 'Enable Article 12 post-market monitoring', endpoint 'Monitoring webhook URL', metrics_export 'none | prometheus | datado…
  severity Critical
  Where. docs/usermanuals/en/reference/configuration.md:280-287 (Turkish mirror: docs/usermanuals/tr/reference/configuration.md:281-288) · type: false claim
  The document says. monitoring: — 'Enable Article 12 post-market monitoring', endpoint 'Monitoring webhook URL', metrics_export 'none | prometheus | datadog | custom_webhook', alert_on_drift 'webhook alert on drift-detector regression', check_interval_hours 'monitoring cadence in hours'.
  What the code does. The monitoring block is declarative metadata only. When enabled=true its values are copied into the training manifest and surfaced in the Annex IV monitoring_and_logging section. No code exports metrics, contacts the endpoint, runs a drift detector, sends an alert, or schedules checks. The only consumers are the manifest writer and the model-card exclusion list.
  Code references. forgelm/compliance.py:1581-1589, forgelm/compliance.py:1856-1857, forgelm/model_card.py:296, forgelm/config.py:1279-1298
  Proposed correction. Add above the block: "monitoring: is declarative metadata: ForgeLM records these values in the compliance manifest and the Annex IV monitoring_and_logging section so your technical documentation names your monitoring setup. ForgeLM itself does not export metrics, contact the endpoint, detect drift, send alerts or run scheduled checks — operate…
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BLEGAL-048 (docs, High) — site/js/translations.js: false claim — [IS-T09-19] 'Post-market monitoring (Article 12+17) wires drift detection into Prometheus / Datadog after deployment.' (TR :2076, DE :2471, FR :2866, ES :3261,…
  severity High
  Where. site/js/translations.js:1681 · type: false claim
  The document says. [IS-T09-19] 'Post-market monitoring (Article 12+17) wires drift detection into Prometheus / Datadog after deployment.' (TR :2076, DE :2471, FR :2866, ES :3261, :3656)
  What the code does. monitoring.metrics_export (forgelm/config.py:1291) has no exporter, and no drift detector exists. Prometheus pulls by scraping (the Pushgateway takes the exposition format, not JSON), and Datadog takes DogStatsD, /api/v2/series or OTel; neither accepts ForgeLM's webhook JSON. Legally, the post-market monitoring system and plan are AI Act Art. 72, reflected in the QMS at Art. 17(1)(h). Art. 12 onl…
  Proposed correction. Monitoring (monitoring.*) records an endpoint for your own post-market monitoring stack (AI Act Art. 72); ForgeLM does not export metrics or detect drift. Apply the same wording to all six locales.

• DOC-D1B05-003 (docs, High) — docs/qms/sop_incident_response.md: false claim — Incidents may be detected by: Runtime monitoring alerts (if monitoring.alert_on_drift: true)
  severity High
  Where. docs/qms/sop_incident_response.md:31 (Turkish mirror: docs/qms/sop_incident_response-tr.md:31) · type: false claim
  The document says. Incidents may be detected by: Runtime monitoring alerts (if monitoring.alert_on_drift: true)
  What the code does. ForgeLM has no drift detector and emits no monitoring alert. monitoring.alert_on_drift is only copied into the compliance manifest (when monitoring.enabled), although its field description says 'Emit a webhook alert when drift detector flags a regression'.
  Code references. forgelm/config.py:1295-1297, forgelm/compliance.py:1582-1589
  Proposed correction. Fix the code (implement or remove the alert behaviour and correct the field description). Interim doc text: "Runtime monitoring alerts from your own serving/monitoring stack (ForgeLM records monitoring.* settings in the compliance manifest but does not itself detect drift or emit alerts)."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D4B02-003 (docs, High) — config_template.yaml: misleading — Post-Market Monitoring block (enabled, endpoint_env, metrics_export: prometheus, alert_on_drift: true, check_interval_hours: 24) presented as an active monitoring f…
  severity High
  Where. config_template.yaml:258-264 · type: misleading
  The document says. Post-Market Monitoring block (enabled, endpoint_env, metrics_export: prometheus, alert_on_drift: true, check_interval_hours: 24) presented as an active monitoring feature
  What the code does. The monitoring block is metadata only. compliance.py copies endpoint, metrics_export, alert_on_drift and check_interval_hours into the compliance manifest (Annex IV §6). No code exports metrics, polls on an interval, detects drift or sends alerts, and endpoint_env is never resolved to send anything.
  Code references. forgelm/compliance.py:1581-1589, forgelm/compliance.py:1856-1857, forgelm/config.py:1279-1298
  Proposed correction. Fix the code, or narrow the promise. Temporary doc note: "# Recorded in the Annex IV manifest only — ForgeLM does not export metrics, poll, or send drift alerts; wire your own monitoring stack."

• DOC-D4B09-008 (docs, High) — site/js/translations.js: false claim — 'Audit log entries record every dispatch — including failures — so you have an independent trail of what was sent'
  severity High
  Where. site/js/translations.js:1778 (wizard.webhook.enabled.detail) · type: false claim
  The document says. 'Audit log entries record every dispatch — including failures — so you have an independent trail of what was sent'
  What the code does. No audit event is emitted for webhook dispatches. The only log_event emitters are compliance/pipeline/safety/model events, and the audit catalog's webhook section only correlates webhook events to audit events (docs/reference/audit_event_catalog.md:150-170). Failed or successful posts appear only in the process log.
  Code references. forgelm/webhook.py:102-240, docs/reference/audit_event_catalog.md:150-170
  Proposed correction. Delete the sentence, or write: 'Delivery failures are logged as warnings in the process log only; the audit log records the underlying training events (training.started, model.reverted, human_approval.required, and so on) that the webhooks mirror.'

• DOC-D4B09-009 (docs, High) — site/js/translations.js: false claim — Post-market monitoring 'wires training metrics into Prometheus / Datadog / a custom webhook'; 'wires the right hooks into the training run so your existing obse…
  severity High
  Where. site/js/translations.js:1783-1796 (wizard.monitoring.hint/.enabled.hint/.enabled.detail/.endpoint.hint/.alert_on_drift.hint), :1681 (wizard.step.operations.tutorial) · type: false claim
  The document says. Post-market monitoring 'wires training metrics into Prometheus / Datadog / a custom webhook'; 'wires the right hooks into the training run so your existing observability stack picks them up'; alert_on_drift 'emits a webhook alert when the drift detector flags a regression'; 'wires drift detection i…
  What the code does. The monitoring block is only copied into the compliance manifest (forgelm/compliance.py:1581-1589). No code pushes metrics to the endpoint, exports Prometheus or Datadog metrics, runs a drift detector, honours check_interval_hours, or sends a drift alert. Grep finds no consumer of metrics_export, alert_on_drift or the endpoint outside config.py and compliance.py.
  Code references. forgelm/compliance.py:1581-1589, forgelm/config.py:1279-1298
  Proposed correction. Describe it as: 'Declares your post-market monitoring endpoint and settings in the Annex IV manifest. ForgeLM does not push metrics or detect drift itself; wire your own observability stack.' Drop 'drift detector' wording.

• DOC-BFIELD-017 (docs, Medium) — forgelm/config.py: false claim — Enable Article 12 post-market monitoring hooks; monitoring-system webhook URL; metrics exporter prometheus/datadog/custom_webhook; emit a webhook alert when drift det…
  severity Medium
  Where. forgelm/config.py:1284-1298 (MonitoringConfig.enabled/endpoint/metrics_export/alert_on_drift/check_interval_hours, F23 F24 F26 F27 F28) · type: false claim
  The document says. Enable Article 12 post-market monitoring hooks; monitoring-system webhook URL; metrics exporter prometheus/datadog/custom_webhook; emit a webhook alert when drift detector flags a regression; monitoring check cadence in hours
  What the code does. The only runtime consumer is compliance.py:1582-1589. It copies the fields into manifest['monitoring'] when enabled is true. No exporter, no POST to the endpoint, no drift detector and no scheduled check exist anywhere in forgelm/. A grep for drift finds only data-audit schema drift. The block is declarative metadata for the Annex IV §6 section (compliance.py:1857).
  Code references. forgelm/compliance.py:1581-1589, forgelm/compliance.py:1856-1857
  Proposed correction. Reword every MonitoringConfig description as 'recorded in the compliance manifest / Annex IV §6; ForgeLM does not run monitoring'. Example: alert_on_drift = 'Declared in the manifest: whether the deployer's monitoring system alerts on drift.'

• DOC-D3B04A-014 (docs, Medium) — docs/usermanuals/en/reference/configuration.md: false claim — endpoint_env: null # env var overriding endpoint; '*_env fields name an environment variable that is read directly via os.environ'.
  severity Medium
  Where. docs/usermanuals/en/reference/configuration.md:283 (and :193 'read directly via os.environ') (Turkish mirror: docs/usermanuals/tr/reference/configuration.md:284 (and :193)) · type: false claim
  The document says. endpoint_env: null # env var overriding endpoint; '*_env fields name an environment variable that is read directly via os.environ'.
  What the code does. The manifest records endpoint or "$" + endpoint_env, so endpoint wins whenever it is set. The variable is never read; only the literal string '$NAME' is stored. The schema description ('overrides endpoint when set') contradicts the implementation.
  Code references. forgelm/compliance.py:1585, forgelm/config.py:1288-1290
  Proposed correction. "endpoint_env: null # name of an env var; recorded as \"$NAME\" in the manifest only when endpoint is empty (the variable is not read)". Also qualify :193: "… except monitoring.endpoint_env, which is recorded by name only."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BFIELD-031 (docs, Low) — forgelm/config.py: wrong detail — Env var name carrying the endpoint URL (overrides endpoint when set).
  severity Low
  Where. forgelm/config.py:1288-1290 (MonitoringConfig.endpoint_env, F25) (Turkish mirror: docs/usermanuals/en/reference/configuration.md:283) · type: wrong detail
  The document says. Env var name carrying the endpoint URL (overrides endpoint when set).
  What the code does. compliance.py:1585 writes mon_cfg.endpoint or f"${mon_cfg.endpoint_env}", so endpoint wins and endpoint_env is never resolved. Reproduced: with both fields set, the manifest endpoint was the literal URL. With only endpoint_env set, the manifest recorded the string '$MON_URL'. When both fields are empty it records '$None'.
  Code references. forgelm/compliance.py:1585
  Proposed correction. 'Env var name recorded as the endpoint reference when endpoint is empty (not resolved).'

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}