--compliance-export always writes empty-metrics, run_id-less artefacts that fail verify-annex-iv and can overwrite a real run's evidence (CR-A04-001) #371

Wave 1 · theme D1-2: Audit chain, integrity manifest and verifier robustness · highest severity High · 4 finding(s) · effort small

Summary

A compliance officer regenerates the Annex IV bundle with --compliance-export after training, as the help advises. Exit 0, but the document is non-compliant and has no metrics.

Primary finding

CR-A04-001 — --compliance-export always writes empty-metrics, run_id-less artefacts that fail verify-annex-iv and can overwrite a real run's evidence

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. _run_compliance_export (forgelm/cli/_no_train_modes.py:248-261) calls generate_training_manifest(config, metrics={}) unconditionally, with no run_id or gate results, and never reads the trained run's output. build_annex_iv_artifact copies the empty dict into performance_metrics, which verify-annex-iv rejects (exit 1). The export writes into the target dir with no existing-artefact check, so pointing it at a run's compliance/ (as the 'run after training' help invites) replaces the real compliance_report.json/annex_iv_metadata.json. The --output flag is never read.

Where.

• forgelm/cli/_no_train_modes.py:243-257 (_run_compliance_export; the metrics={} call is at line 248 of the cited range)
• forgelm/cli/_parser.py:1454-1460
• forgelm/cli/_no_train_modes.py:248
• forgelm/cli/_no_train_modes.py:253
• forgelm/cli/_parser.py:1450
• forgelm/cli/_parser.py:1455
• forgelm/compliance.py:1774
• forgelm/compliance.py:1859 (performance_metrics copies the empty evaluation_results.metrics). Line 1783 is only the build_annex_iv_artifact docstring, so the cited lo…
• forgelm/verify/_annex_iv.py:42
• docs/guides/safety_compliance.md:488-495

Evidence.

• cexp-secret/annex_iv_metadata.json: verify-annex-iv exits 1 with 'Missing or empty required Annex IV field(s): performance_metrics'. training_manifest.yaml has final_metrics: {} although out-secret had already trained. The in-run compliance/annex_iv_metadata.json verifies 0.
• single/ce: after a training run, --compliance-export ./out/compliance EXIT=0. Original compliance_report.json had evaluation_results.metrics full and run_id fg-0f030e8624ee. Afterwards evaluation_results {'metrics': {}} and run_id None. No warning in stderr.
• fpy -m forgelm --config cfg_c.yaml --compliance-export out/ce EXIT=0 (5 files), then fpy -m forgelm verify-annex-iv out/ce/annex_iv_metadata.json -> 'Missing or empty required Annex IV field(s): performance_metrics.' EXIT=1.

Recommended fix. Load metrics from the run (for example compliance/compliance_report.json under training.output_dir) when present, or fix the help and refuse/warn that the output will not pass verify-annex-iv.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

DOC-BHELP-003 — forgelm: false claim — --output DIR: 'Output directory for --compliance-export (default: ./compliance).'

severity Medium

Where. forgelm/cli/_parser.py:1459 (--output) · type: false claim

The document says. --output DIR: 'Output directory for --compliance-export (default: ./compliance).'

What the code does. --compliance-export OUTPUT_DIR itself takes the destination and --output is ignored by it. There is no ./compliance default. Verified: '--compliance-export a --output b' wrote only to a and b was never created. Top-level args.output is read only as the fallback for 'forgelm audit'.

Code references. forgelm/cli/_no_train_modes.py:281-282, forgelm/cli/subcommands/_audit.py:196,347, forgelm/cli/_parser.py:1459

User impact. A user passing --output expects the bundle there and gets it in the --compliance-export directory. A user relying on the ./compliance default gets a required-argument error instead.

Proposed correction. Remove the flag from the help, or document it as the audit fallback. Or make --compliance-export honour --output and add the default.

DOC-D3B04A-009 — docs/usermanuals/en/reference/cli.md: false claim — --output DIR — 'Output directory for --compliance-export (default: ./compliance/).'

severity Medium

Where. docs/usermanuals/en/reference/cli.md:56 (Turkish mirror: docs/usermanuals/tr/reference/cli.md:56) · type: false claim

The document says. --output DIR — 'Output directory for --compliance-export (default: ./compliance/).'

What the code does. --compliance-export takes its own required OUTPUT_DIR argument, and that argument is the only directory used. The top-level args.output is never read by the compliance export, so there is no ./compliance default. Reproduced: --compliance-export ./exportA --output ./exportB wrote all files to ./exportA, ./exportB was never created, EXIT=0. The parser help carries the same false text.

Code references. forgelm/cli/_no_train_modes.py:248-283, forgelm/cli/_parser.py:1442-1459

User impact. Operators passing --output to redirect the evidence bundle find it in a different directory, silently. A compliance bundle can land outside the controlled location.

Proposed correction. Delete the --output DIR row, or reword it: "--output DIR — accepted at top level but ignored by --compliance-export; the export directory is the OUTPUT_DIR argument of --compliance-export itself (no default)."

Turkish mirror. the Turkish page has the same error; fix both in the same PR

DOC-E2E-001 — forgelm/cli/_parser.py: false claim — --compliance-export: 'Run after training so the manifest is complete; standalone use produces artifacts with empty metrics.'

severity Medium

Where. forgelm/cli/_parser.py:1450-1451 · type: false claim

The document says. --compliance-export: 'Run after training so the manifest is complete; standalone use produces artifacts with empty metrics.'

What the code does. metrics are always {}, and the Annex IV fails verify-annex-iv (exit 1)

Code references. forgelm/cli/_no_train_modes.py:253

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}