Code does not match docs/usermanuals/en/getting-started/first-run.md: The run produces checkpoints/customer-support/artifacts/ with annex_iv_metadata.json, audit_log.jsonl, data… (CR-A02-023) #96

Wave 0 · theme D0-5: Documented controls that are not implemented · highest severity Critical · 2 finding(s) · effort not estimated

Summary

The document states: The run produces checkpoints/customer-support/artifacts/ with annex_iv_metadata.json, audit_log.jsonl, data_audit_report.json, safety_report.json, benchmark_results.json and manifest.json ('SHA-256 over every artifact'), plus README.md, config_snapshot.yaml and adapter_model.safetensors at the run root. 'Every file inside is hashed in manifest.json for tamper-evidence.'. Observed in the code: There is no artifacts/ directory and no manifest.json over the compliance files. Compliance files go to <output_dir>/compliance/ (forgelm/trainer.py:2087-2088): compliance_report.json, training_manifest.yaml, data_provenance.json, an optional risk_assessment.json, annex_iv_metadata.json (only when compliance.provider_name, system_name or intended_purpose is set; forgelm/compliance.py:1804-1815) and data_governance_report.json. audit_log.jsonl and its .manifest.json genesis pin sit at the output_dir root (forgelm/compliance.py:175-176). Safety and benchmark results go to <output_dir>/safety/safety_results.json and <output_dir>/benchmark/benchmark_results.json, and only when enabled (forgelm/trainer.py:1776,1931; forgelm/safety/_results.py:106). README.md, mod…

Primary finding

CR-A02-023 — Code does not match docs/usermanuals/en/getting-started/first-run.md: The run produces checkpoints/customer-support/artifacts/ with annex_iv_metadata.json, audit_log.jsonl, data_audit_report.json,…

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed

What happens. The tamper-evidence statement is a C1 integrity promise (compare F-088 / F-083). The code provides no integrity manifest over the compliance bundle and no verifier for it. Per the authority table this needs code verification: either add a bundle manifest or formally retract the promise.

Where.

• forgelm/trainer.py:2087-2088
• forgelm/compliance.py:175-176
• forgelm/compliance.py:760-813
• forgelm/compliance.py:1811-1815
• forgelm/trainer.py:1776
• forgelm/trainer.py:1931
• forgelm/trainer.py:2158-2166
• forgelm/model_card.py:342
• docs/usermanuals/en/getting-started/first-run.md:162-176
• docs/usermanuals/tr/getting-started/first-run.md:162-176

Evidence.

• forgelm/trainer.py:2087-2088; forgelm/compliance.py:175-176; forgelm/compliance.py:760-813; forgelm/compliance.py:1804-1815; forgelm/trainer.py:1776; forgelm/trainer.py:1931; forgelm/trainer.py:2163; forgelm/model_card.py:342

Recommended fix. Fix the code or the claim: either emit a SHA-256 manifest over compliance/ that a verify-* command can check, or remove 'Every file inside is hashed in manifest.json for tamper-evidence'. Interim doc text: show the real tree (audit_log.jsonl + audit_log.jsonl.manifest.json, compliance/{compliance_report.json, training_manifest.yaml, data_provenance.json, data_governance_report.json, annex_iv_metadata.json*}, safety/, benchmark/, final_model/{adapter_model.safetensors, README.md, model_integrity.json, deployer_instructions.md}; * = only when configured). State that only final_model/ is covered by model_integrity.json (forgelm verify-integrity), and the audit log by its hash chain (forgelm verify-audit).

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

DOC-D3B04B-001 — docs/usermanuals/en/getting-started/first-run.md: false claim — The run produces checkpoints/customer-support/artifacts/ with annex_iv_metadata.json, audit_log.jsonl, data_audit_report.json, safety…

severity Critical

Where. docs/usermanuals/en/getting-started/first-run.md:162-176 (Turkish mirror: docs/usermanuals/tr/getting-started/first-run.md:162-176) · type: false claim

The document says. The run produces checkpoints/customer-support/artifacts/ with annex_iv_metadata.json, audit_log.jsonl, data_audit_report.json, safety_report.json, benchmark_results.json and manifest.json ('SHA-256 over every artifact'), plus README.md, config_snapshot.yaml and adapter_model.safetensors at the run root. 'Every file inside is hashed in manifest.json for tamper-evidence.'

What the code does. There is no artifacts/ directory and no manifest.json over the compliance files. Compliance files go to <output_dir>/compliance/ (forgelm/trainer.py:2087-2088): compliance_report.json, training_manifest.yaml, data_provenance.json, an optional risk_assessment.json, annex_iv_metadata.json (only when compliance.provider_name, system_name or intended_purpose is set; forgelm/compliance.py:1804-1815) and data_governance_report.json. audit_log.jsonl and its .manifest.json genesis pin sit at the output_dir root (forgelm/compliance.py:175-176). Safety and benchmark results go to <output_dir>/safety/safety_results.json and <output_dir>/benchmark/benchmark_results.json, and only when enabled (forgelm/trainer.py:1776,1931; forgelm/safety/_results.py:106). README.md, model_integrity.json and the adapter weights are in <output_dir>/final_model/ (forgelm/model_card.py:342; forgelm/trainer.py:1590-1593,2163). Nothing writes config_snapshot.yaml (no writer in forgelm/). Training never runs a data audit, so data_audit_report.json exists only if the operator put forgelm audit output there. The only SHA-256 manifest is model_integrity.json, and it covers final_model/ alone (forgelm/co…

Code references. forgelm/trainer.py:2087-2088, forgelm/compliance.py:175-176, forgelm/compliance.py:760-813, forgelm/compliance.py:1804-1815

User impact. Users and auditors look for files that do not exist. The tamper-evidence claim makes them believe the whole evidence bundle is integrity-protected when no manifest covers compliance_report.json, data_provenance.json and the rest. An edit to those files cannot be detected by any verifier.

Proposed correction. Fix the code or the claim: either emit a SHA-256 manifest over compliance/ that a verify-* command can check, or remove 'Every file inside is hashed in manifest.json for tamper-evidence'. Interim doc text: show the real tree (audit_log.jsonl + audit_log.jsonl.manifest.json, compliance/{compliance_report.json, training_manifest.yaml, data_provenance.json, data_governance_report.json, annex_iv_metadata.json*}, safety/, benchmark/, final_model/{adapter_model.safetensors, README.md, model_integrity.json, deployer_instructions.md}; * = only when configured). State that only final_model/ is covered by model_integrity.json (forgelm verify-integrity), and the audit log by its hash chain (forgelm verify-audit).

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}