Experiment tracking, webhooks and monitoring: Experiment-tracking user manual and standards text (docs/usermanuals/{en,tr}/operations/experiment-tracking.md; docs/standards/logging-observability.md)… (TECH-T09-004) #361

Wave 1 · theme D1-11: Other high-severity packages · highest severity High · 6 finding(s) · effort not estimated

Summary

A user who follows line 52 installs forgelm[tracking], still has no mlflow, and the run dies at trainer construction (the failure pyproject.toml:95-99 describes). Line 65 points to a flag that does not exist, and lines 78-79 promise metrics that never appear.

Primary finding

TECH-T09-004 — Experiment tracking, webhooks and monitoring: Experiment-tracking user manual and standards text (docs/usermanuals/{en,tr}/operations/experiment-tracking.md; docs/standards/logging-observability.md)…

severity Next release

Item. Experiment-tracking user manual and standards text (docs/usermanuals/{en,tr}/operations/experiment-tracking.md; docs/standards/logging-observability.md) · verdict: incorrect-claim · urgency: Next release

State in ForgeLM. experiment-tracking.md (EN and TR, same line numbers) contains four errors. Line 52: 'MLflow requires the [tracking] extra', but pyproject.toml:101-102 puts mlflow in [tracking-mlflow] and [tracking] holds only wandb (:91-93). Line 61: tensorboard is 'bundled with PyTorch >= 1.4' and must be installed separately, but it is a ForgeLM core dependency (pyproject.toml:60) and torch's requirements do not include it (checked in the lower venv). Line 65: 'ForgeLM's --report-to CLI override', but rg finds no report-to flag anywhere in forgelm/. Lines 78-79: system/gpu_utilization and system/vram_used_gb 'Sampled every 30s', but rg finds no emitter in forgelm/. In docs/standards/logging-observability.md:198, 'tracking.wandb.enabled: true' is not a config key.

State of the ecosystem on 2026-10-01. These are internal factual errors checked against the snapshot. The upstream behaviour is current: transformers v5.18.0 needs tensorboard, wandb or mlflow importable for the callbacks, and tensorboard is a separate PyPI package.

Impact. A user who follows line 52 installs forgelm[tracking], still has no mlflow, and the run dies at trainer construction (the failure pyproject.toml:95-99 describes). Line 65 points to a flag that does not exist, and lines 78-79 promise metrics that never appear.

Recommendation. Fix the EN and TR mirrors together (bilingual parity guard): MLflow -> [tracking-mlflow]; drop the PyTorch-bundled sentence; remove the --report-to paragraph or implement a list-valued override; remove the system/* rows or implement them; fix logging-observability.md:198 to name training.report_to.

Sources.

• transformers v5.18.0 integration_utils (callbacks require the packages importable) (2026-09-30)
• PyPI tensorboard metadata (independent package) (2026-06-29)
• PyPI transformers metadata (5.18.0) (2026-09-30)

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• DOC-D1B07-004 (docs, High) — docs/standards/logging-observability.md: broken example — Tracking is 'Opt-in via config (tracking.wandb.enabled: true), never by default.'
  severity High
  Where. docs/standards/logging-observability.md:198 · type: broken example
  The document says. Tracking is 'Opt-in via config (tracking.wandb.enabled: true), never by default.'
  What the code does. No tracking section exists: ForgeConfig forbids extra keys, and --dry-run with tracking: {wandb: {enabled: true}} fails with 'tracking: Extra inputs are not permitted'. The real key is training.report_to (tensorboard|wandb|mlflow|none), and its default is "tensorboard", so TensorBoard reporting is on by default.
  Code references. forgelm/config.py:604-607, forgelm/config.py (ForgeConfig fields: auth, compliance, data, distributed, evaluation, lora, merge, model, monitoring, pipeline, retention, risk_assessment,…
  Proposed correction. Replace with: 'Selected via training.report_to (tensorboard default, wandb [tracking extra], mlflow [tracking-mlflow extra], none to disable).'

• DOC-D3B06-001 (docs, High) — docs/usermanuals/en/operations/experiment-tracking.md: wrong detail — "MLflow requires the [tracking] extra."
  severity High
  Where. docs/usermanuals/en/operations/experiment-tracking.md:52 (Turkish mirror: docs/usermanuals/tr/operations/experiment-tracking.md:52) · type: wrong detail
  The document says. "MLflow requires the [tracking] extra."
  What the code does. MLflow lives in the separate tracking-mlflow extra. tracking installs only wandb (pyproject.toml:87-103). The schema description agrees (config.py:606).
  Code references. pyproject.toml:87-103, forgelm/config.py:604-607
  Proposed correction. "MLflow requires the [tracking-mlflow] extra: pip install 'forgelm[tracking-mlflow]'."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B06-002 (docs, High) — docs/usermanuals/en/operations/experiment-tracking.md: false claim — "To stream to multiple backends in the same run, use ForgeLM's --report-to CLI override." The tip says to "Configure both" Tenso…
  severity High
  Where. docs/usermanuals/en/operations/experiment-tracking.md:65,146 (Turkish mirror: docs/usermanuals/tr/operations/experiment-tracking.md:65) · type: false claim
  The document says. "To stream to multiple backends in the same run, use ForgeLM's --report-to CLI override." The tip says to "Configure both" TensorBoard and W&B.
  What the code does. No --report-to flag exists (forgelm --help). forgelm --config X --dry-run --report-to wandb exits 2 with an argparse error (this audit). training.report_to is a single Literal (forgelm/config.py:604-607), forwarded as-is (trainer.py:519). ForgeLM has no way to report to more than one backend.
  Code references. forgelm/config.py:604-607, forgelm/trainer.py:519, forgelm/cli/_parser.py:1340-1460
  Proposed correction. "ForgeLM reports to exactly one backend per run (training.report_to); there is no CLI override." Drop the "Use multiple backends in parallel" tip.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BLEGAL-232 (docs, Low) — docs/usermanuals/en/operations/experiment-tracking.md: false claim — [IS-T09-24] 'system/gpu_utilization | Sampled every 30s' / 'system/vram_used_gb | Sampled every 30s'
  severity Low
  Where. docs/usermanuals/en/operations/experiment-tracking.md:78-79 (Turkish mirror: docs/usermanuals/tr/operations/experiment-tracking.md:78-79) · type: false claim
  The document says. [IS-T09-24] 'system/gpu_utilization | Sampled every 30s' / 'system/vram_used_gb | Sampled every 30s'
  What the code does. No code in forgelm/ emits these metrics (internal factual error checked against the snapshot; the upstream tracker integration is current).
  Proposed correction. Remove the two system/* rows (or note that system metrics come from the tracker's own agent, e.g. W&B system monitoring, not ForgeLM).
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B06-031 (docs, Low) — docs/usermanuals/en/operations/experiment-tracking.md: misleading — TensorBoard "requires installing either tensorboard (bundled with PyTorch ≥ 1.4) or tensorboardX separately".
  severity Low
  Where. docs/usermanuals/en/operations/experiment-tracking.md:61 (Turkish mirror: docs/usermanuals/tr/operations/experiment-tracking.md:61) · type: misleading
  The document says. TensorBoard "requires installing either tensorboard (bundled with PyTorch ≥ 1.4) or tensorboardX separately".
  What the code does. tensorboard>=2.15.0,<3.0.0 is a core ForgeLM dependency (pyproject.toml:60), so the default backend works after a plain install. PyTorch does not bundle tensorboard.
  Code references. pyproject.toml:45-68
  Proposed correction. "tensorboard is installed with ForgeLM; no extra step is needed."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• Each statement is re-checked against the code at fix time.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}