Failed or reverted runs produce no compliance bundle or model card, although docs and site promise one after every successful or failed run (CR-A02-010) #431

Wave 1 · theme D1-8: Docs describing features that do not exist (doc side of code findings) · highest severity High · 5 finding(s) · effort medium

Summary

A regulator asks for the Article 11 record of a rejected model; only the audit log exists.

Primary finding

CR-A02-010 — Failed or reverted runs produce no compliance bundle or model card, although docs and site promise one after every successful or failed run

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. Compliance export, the model card and integrity/deployer artefacts are produced only by _finalize_artifacts (trainer.py:1496-1506), reached at trainer.py:1652 after every gate; each gate-failure branch of _run_training_pipeline returns earlier (trainer.py:1622-1650) and an exception path only emits pipeline.failed, contradicting site/compliance.html:268 and the compliance overview.

Where.

• forgelm/trainer.py:1622-1652 (early returns before _finalize_artifacts)
• site/compliance.html:268
• docs/usermanuals/en/compliance/overview.md:8
• docs/usermanuals/en/getting-started/project-layout.md:128 (also names a non-existent artifacts/ directory)
• site/index.html:184-185 and 370-371
• forgelm/trainer.py:1496-1506 (_finalize_artifacts)
• forgelm/trainer.py:1622
• forgelm/trainer.py:1652 (sole _finalize_artifacts call)
• site/index.html:185
• forgelm/trainer.py:1496-1506 (confirmed)

Evidence.

• r4: env/scripted/configs/train_eval_gates.yaml with min_score 0.9 and scenario bench-valid -> EXIT=3; out/ has audit_log.jsonl, benchmark/, checkpoint-1, no compliance/ directory and no model card; audit ends with model.reverted {reason: benchmark}.
• Stubbed CLI run with an auto-revert loss breach: EXIT 3, _export_compliance_if_needed calls = [], no compliance dir. Events: pipeline.initialized, training.started, evaluation.loss_gate_completed, model.reverted.
• runB and runC (auto-reverted loss gate): no compliance/ directory. runA (safety no-evidence) and runF (loss breach with auto_revert=false): compliance/ with compliance_report.json, data_governance_report.json, data_provenance.json, training_manifest.yaml, and final_model/ with model_integrity.json and deployer_instructions.md.

Recommended fix. Either export the compliance manifest (with the failed gate results) before reverting, or correct the docs and site to say the bundle is produced for promoted runs only.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• DOC-D2B06-008 (docs, High) — docs/guides/enterprise_deployment.md: false claim — 'Every training run automatically generates compliance artifacts in checkpoints/compliance/' … 'no additional configuration needed'. The Full Evide…
  severity High
  Where. docs/guides/enterprise_deployment.md:199-227 · type: false claim
  The document says. 'Every training run automatically generates compliance artifacts in checkpoints/compliance/' … 'no additional configuration needed'. The Full Evidence Bundle tree places risk_assessment.json, data_governance_report.json, annex_iv_metadata.json, deployer_instructions.md, model_integrity.json and aud…
  What the code does. (1) compliance/ is written only on the success path, after every gate passes. A reverted run produced no compliance/ directory (reproduced on tiny-qwen). (2) annex_iv_metadata.json is written only when a compliance: block is configured, and risk_assessment.json only with risk_assessment:. A run with this guide's production template produced neither. (3) deployer_instructions.md and model_inte…
  Code references. forgelm/trainer.py:1621-1658, forgelm/trainer.py:1496-1506, forgelm/compliance.py:1551-1572, forgelm/compliance.py:2201-2236
  Proposed correction. Fix the code so failed or reverted runs still export the compliance bundle (as the project promises). Correct the tree to: checkpoints/{audit_log.jsonl, audit_log.jsonl.manifest.json, compliance/{compliance_report.json, training_manifest.yaml, data_provenance.json, data_governance_report.json, risk_assessment.json*, annex_iv_metadata.json*}, final…

• DOC-D3B01-007 (docs, High) — docs/usermanuals/en/compliance/overview.md: false claim — Every successful (or failed) run produces a structured evidence bundle that maps cleanly onto EU AI Act Articles 9-17 ...
  severity High
  Where. docs/usermanuals/en/compliance/overview.md:8 (Turkish mirror: docs/usermanuals/tr/compliance/overview.md:8) · type: false claim
  The document says. Every successful (or failed) run produces a structured evidence bundle that maps cleanly onto EU AI Act Articles 9-17 ...
  What the code does. A run that fails a gate under auto_revert returns before _finalize_artifacts, so it gets no compliance/ directory, no model card and no model_integrity.json. A crashed run logs pipeline.failed and re-raises. Repro: scripted evalloss-nan with auto_revert:true exited 3. The output dir held only audit_log.jsonl (+ manifest), checkpoint-1, runs/ and the HF Trainer README. Audit events: pipeline.initi…
  Code references. forgelm/trainer.py:1622-1652, forgelm/trainer.py:1498-1506, forgelm/trainer.py:1700-1709, review-probe:reverted_run_evalloss_nan.log
  Proposed correction. Fix the code: write the compliance bundle (compliance_report.json with the failing gate verdict, data_provenance.json, data_governance_report.json) on gate failure and crash too. Until then reword: "Every successful run produces a structured evidence bundle ...; a run that fails a gate (auto-revert) or crashes leaves only the audit log (`model.rev…
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B04B-007 (docs, High) — docs/usermanuals/en/getting-started/project-layout.md: false claim — A run writes checkpoints/<name>/{adapter_model.safetensors, README.md, config_snapshot.yaml, artifacts/}. 'Every successful (or…
  severity High
  Where. docs/usermanuals/en/getting-started/project-layout.md:23-28,53,126-132 (Turkish mirror: docs/usermanuals/tr/getting-started/project-layout.md:25-28,53,126-129) · type: false claim
  The document says. A run writes checkpoints/<name>/{adapter_model.safetensors, README.md, config_snapshot.yaml, artifacts/}. 'Every successful (or failed) training run creates a checkpoints//artifacts/ directory'. Mixing runs 'breaks the SHA-256 manifest'.
  What the code does. Weights, README.md and model_integrity.json go to <output_dir>/final_model/ (forgelm/trainer.py:1590-1593; forgelm/model_card.py:342). The evidence bundle goes to <output_dir>/compliance/ (forgelm/trainer.py:2087), and audit_log.jsonl to the output_dir root (forgelm/compliance.py:175). Nothing writes config_snapshot.yaml. The bundle is exported only from _finalize_artifacts, which runs after…
  Code references. forgelm/trainer.py:1590-1593, forgelm/trainer.py:1640-1652, forgelm/trainer.py:2087, forgelm/compliance.py:175
  Proposed correction. Show checkpoints/<name>/{audit_log.jsonl, compliance/, final_model/{adapter_model.safetensors, README.md, model_integrity.json}}. Say 'every run that completes its gates writes compliance/; failed or reverted runs leave only audit_log.jsonl'. Drop config_snapshot.yaml and the 'SHA-256 manifest' sentence, or point it at model_integrity.json (…
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D4B08-004 (docs, High) — site/compliance.html: false claim — After every successful (or failed) run, this is what lands in your output directory (compliance/ bundle, final_model/README.md, deployer_instructions.md, model_int…
  severity High
  Where. site/compliance.html:268 (and translations.js:3888); compliance.header.lead translations.js:3848 'Every run produces the artifacts' · type: false claim
  The document says. After every successful (or failed) run, this is what lands in your output directory (compliance/ bundle, final_model/README.md, deployer_instructions.md, model_integrity.json).
  What the code does. The model card, model_integrity.json, deployer_instructions.md and the whole compliance/ bundle are written only in _finalize_artifacts (forgelm/trainer.py:1496-1506). The trainer reaches it at trainer.py:1652 only after the eval-loss, benchmark, safety and judge steps all return True. An eval-loss gate failure (trainer.py:1622-1639), an auto-reverted gate (1641-1650) or a training exception retu…
  Code references. forgelm/trainer.py:1496-1506, forgelm/trainer.py:1622-1652
  Proposed correction. Fix the code: export the compliance bundle on failed and reverted runs as well. Until then, the temporary doc text could read: 'After every successful run (and after a run whose failed gate was kept because auto_revert is off), this is what lands... Failed or reverted runs leave only audit_log.jsonl and its manifest.'

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}