model_integrity.json is written before deployer_instructions.md, so verify-integrity exits 6 on every freshly trained model (CR-A02-002) #370

Wave 1 · theme D1-2: Audit chain, integrity manifest and verifier robustness · highest severity High · 10 finding(s) · effort small

Summary

A CI gate 'forgelm verify-integrity final_model' after training is permanently red with the tamper code. Operators learn to ignore exit 6 or delete deployer_instructions.md, which defeats the signal and the Article 13 artefact.

Primary finding

CR-A02-002 — model_integrity.json is written before deployer_instructions.md, so verify-integrity exits 6 on every freshly trained model

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. ForgeTrainer._finalize_artifacts (forgelm/trainer.py:1503-1505) calls _generate_model_integrity(final_path) and then _generate_deployer_instructions(final_path), which writes deployer_instructions.md into the same final_model directory (compliance.py:1761-1764) after the SHA-256 manifest has been built. verify_integrity exempts only model_integrity.json from its 'added' walk (verify/_model_integrity.py), so an untouched model reports '1 added: deployer_instructions.md' and the CLI exits 6, the documented tamper signal. The Art. 13 document is never covered by the manifest. Some members also note re-run and symlink/permission variants on the same generator/verifier pair.

Where.

• forgelm/trainer.py:1503-1504
• forgelm/trainer.py:2177-2182
• forgelm/compliance.py:1761-1764 (confirmed)
• forgelm/verify/_model_integrity.py:208-217
• forgelm/trainer.py:2157-2187
• forgelm/compliance.py:1681
• forgelm/compliance.py:1761
• forgelm/verify/_model_integrity.py:63-66
• docs/reference/verify_integrity_subcommand.md:34
• forgelm/trainer.py:1503-1506

Evidence.

• verify-integrity out-secret/final_model and out-nosecret/final_model both exit 6 with added=[deployer_instructions.md]. The manifest lists 6 files. Removing deployer_instructions.md (control M0) gives exit 0. The fixture env/artifacts/sft/model_integrity.json, produced by another unit, also lacks deployer_instructions.md.
• verify-integrity exit 6 on sft-dir, sft-file, dpo-dir, dpo-file, kto-dir, kto-file, p-main-simpo, p-main-orpo, p-main-grpo: 'Model artifacts do not match model_integrity.json: 1 added. added: deployer_instructions.md'. The run logs show 'Model integrity checksums saved' before 'Deployer instructions saved'. The shared fixture env/artifacts/sft has the same layout.
• verify-integrity run1/out/final_model gives 'FAIL ... 1 added. added: deployer_instructions.md', exit 6. Same exit 6 on a copy of env/artifacts/sft (ungated).

Recommended fix. Generate deployer instructions (and anything else written into final_model) before the integrity manifest, and add an end-to-end test that runs verify-integrity on a freshly finalised directory.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• DOC-D1B04-014 (docs, High) — docs/reference/verify_integrity_subcommand.md: broken example — Exit 0 when every recorded artefact matches and no unexpected files exist; the example shows a trained final_model verifying OK.
  severity High
  Where. docs/reference/verify_integrity_subcommand.md:31,45,56-59 (Turkish mirror: docs/reference/verify_integrity_subcommand-tr.md:31,58,69) · type: broken example
  The document says. Exit 0 when every recorded artefact matches and no unexpected files exist; the example shows a trained final_model verifying OK.
  What the code does. The trainer writes model_integrity.json before deployer_instructions.md in the same directory (README → integrity → deployer instructions). verify-integrity on a freshly trained final_model, or on an approved staging directory, therefore reports 'added: deployer_instructions.md' and exits 6. Reproduced with the library sequence and the CLI (EXIT=6).
  Code references. forgelm/trainer.py:1503-1506, forgelm/compliance.py:1761, forgelm/verify/_model_integrity.py:197-206
  Proposed correction. Fix the code: generate model_integrity.json after all writers into final_path, or record deployer_instructions.md in it. Temporary note: 'Runs from this version report deployer_instructions.md as added.'
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D1B04-016 (docs, High) — docs/reference/verify_integrity_subcommand.md: broken example — forgelm verify-integrity checkpoints/run/final_model prints 'OK ... All 7 recorded artifact(s) present and unchanged.' on a training…
  severity High
  Where. docs/reference/verify_integrity_subcommand.md:56-58 · type: broken example
  The document says. forgelm verify-integrity checkpoints/run/final_model prints 'OK ... All 7 recorded artifact(s) present and unchanged.' on a training output
  What the code does. Every fresh final_model from a training run exits 6 with 'added: deployer_instructions.md', because the manifest is written before that file (finding F1)
  Code references. forgelm/trainer.py:1504-1505
  Proposed correction. Fix F1. The example then holds.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B01-010 (docs, High) — docs/usermanuals/en/compliance/verify-integrity.md: false claim — model_integrity.json records the SHA-256 of every artifact; a clean exit is the pre-deploy check; run it as a CI/CD release gate and…
  severity High
  Where. docs/usermanuals/en/compliance/verify-integrity.md:8,12,14,119 (Turkish mirror: docs/usermanuals/tr/compliance/verify-integrity.md:8,12,14,119) · type: false claim
  The document says. model_integrity.json records the SHA-256 of every artifact; a clean exit is the pre-deploy check; run it as a CI/CD release gate and fail on non-zero.
  What the code does. The trainer writes model_integrity.json before deployer_instructions.md, so every ForgeLM-trained final_model holds an unrecorded file and verify-integrity exits 6 ("added: deployer_instructions.md"). Repro: the review's real 1-step runs env/artifacts/sft, dpo and kto each exit 6; the fresh scripted run exits 6; the replay script exits 6.
  Code references. forgelm/trainer.py:1503-1505, forgelm/trainer.py:2157-2167, forgelm/compliance.py:1761-1765, forgelm/verify/_model_integrity.py:197-223
  Proposed correction. Fix the code: generate model_integrity.json last in _finalize_artifacts (after the deployer instructions). Until then warn: "Models trained by this version report added: deployer_instructions.md (exit 6) because that file is written after the manifest."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B03-007 (docs, High) — docs/usermanuals/en/reference/json-output.md: false claim — Exit 0 means all recorded artefacts are present and unchanged. Exit 6 means the deployed weights are not the weights that were signed off.…
  severity High
  Where. docs/usermanuals/en/reference/json-output.md:809-822,859 (Turkish mirror: docs/usermanuals/tr/reference/json-output.md:809-863) · type: false claim
  The document says. Exit 0 means all recorded artefacts are present and unchanged. Exit 6 means the deployed weights are not the weights that were signed off. The success example shows a clean model directory.
  What the code does. Every unmodified final_model that ForgeLM produces fails with exit 6, reason '1 added: deployer_instructions.md'. The trainer writes model_integrity.json before deployer_instructions.md, so that file is never recorded. I reproduced this on an untouched copy of a fresh tiny run and on the shared fixtures env/artifacts/sft and env/artifacts/dpo, which all returned valid:false, added:['deployer_inst…
  Code references. forgelm/trainer.py:1503-1505, forgelm/verify/_model_integrity.py:218
  Proposed correction. Fix the code: generate deployer_instructions.md (and anything else written into final_model/) before model_integrity.json, or regenerate the manifest last. Until then, add a warning to this page: "Models trained up to this release report deployer_instructions.md under added and exit 6."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D4B01-008 (docs, High) — README.md: false claim — final_model/model_integrity.json is the SHA-256 of every artefact in this directory
  severity High
  Where. README.md:78 · type: false claim
  The document says. final_model/model_integrity.json is the SHA-256 of every artefact in this directory
  What the code does. deployer_instructions.md is written after the manifest and is not covered, so verify-integrity reports it as added (exit 6)
  Code references. forgelm/trainer.py:1503-1505, forgelm/compliance.py:1761
  Proposed correction. Fix the generation order (code)

• DOC-D4B01-009 (docs, High) — README.md: false claim — final_model/model_integrity.json = "SHA-256 of every artefact in this directory"; forgelm verify-integrity "detects changed, removed, or added artefacts" and exits 6 on m…
  severity High
  Where. README.md:78; README.md:120 · type: false claim
  The document says. final_model/model_integrity.json = "SHA-256 of every artefact in this directory"; forgelm verify-integrity "detects changed, removed, or added artefacts" and exits 6 on mismatch
  What the code does. _finalize_artifacts writes model_integrity.json before deployer_instructions.md, so the Art. 13 document is never hashed. A freshly trained final_model/ therefore fails its own verifier. Probe probes/integrity_order.py ran the real _finalize_artifacts, then forgelm verify-integrity printed "Model artifacts do not match model_integrity.json: 1 added. added: deployer_instructions.md" and exit…
  Code references. forgelm/trainer.py:1496-1506, forgelm/compliance.py:1761-1764, forgelm/compliance.py:760-817, forgelm/verify/_model_integrity.py:197-222
  Proposed correction. Fix the code: generate model_integrity.json last (after the model card and deployer instructions). Temporary note in the README: "Until fixed, a fresh final_model/ reports deployer_instructions.md as added."

• DOC-D1B04-029 (docs, Medium) — docs/reference/compliance_summary.md: misleading — Post-deployment Art. 15 verification via forgelm verify-integrity.
  severity Medium
  Where. docs/reference/compliance_summary.md:34-36 (Turkish mirror: docs/reference/compliance_summary-tr.md:35) · type: misleading
  The document says. Post-deployment Art. 15 verification via forgelm verify-integrity.
  What the code does. verify-integrity on a freshly trained final_model exits 6, because deployer_instructions.md is written after model_integrity.json.
  Code references. forgelm/trainer.py:1503-1506, forgelm/compliance.py:1761
  Proposed correction. Fix the code (generate the manifest last); temporary note: 'verify-integrity currently reports deployer_instructions.md as added.'
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D1B04-045 (docs, Medium) — docs/reference/verify_integrity_subcommand.md: false claim — Exit 0 / 'OK: ... All N recorded artifact(s) present and unchanged' on an untampered training output
  severity Medium
  Where. docs/reference/verify_integrity_subcommand.md:31,55-58 · type: false claim
  The document says. Exit 0 / 'OK: ... All N recorded artifact(s) present and unchanged' on an untampered training output
  What the code does. A fresh final_model exits 6 with added deployer_instructions.md
  Code references. forgelm/trainer.py:1503-1505
  Turkish mirror. the Turkish page has no such statement

• DOC-D4B01-018 (docs, Medium) — README.md: false claim — model_integrity.json — SHA-256 of every artefact in this directory
  severity Medium
  Where. README.md:78 · type: false claim
  The document says. model_integrity.json — SHA-256 of every artefact in this directory
  What the code does. deployer_instructions.md is written after the manifest and is not listed
  Code references. forgelm/trainer.py:1503-1505

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}