[Wave 3] Medium — Docs: docs/guides/safety_compliance.md #579

Wave 3 · 7 finding(s) in 7 fix package(s). One pass over this document; keep the Turkish mirror in sync.

Findings

• DOC-BLEGAL-081 (docs, Medium) — docs/guides/safety_compliance.md: false claim — It requires: - Documented AI inventories - Machine-readable compliance evidence - Training data provenance tracking - Risk classification - Continuous…
  severity Medium
  Where. docs/guides/safety_compliance.md:246-250 (Turkish mirror: docs/guides/safety_compliance-tr.md:279-283) · type: false claim
  The document says. It requires: - Documented AI inventories - Machine-readable compliance evidence - Training data provenance tracking - Risk classification - Continuous monitoring
  What the code does. The term 'AI inventories' does not occur in the Regulation. The closest duty is Art. 49 registration in the EU database for specific providers and deployers. 'Machine-readable' appears only for Art. 50(2) marking of synthetic output, the digital CE marking (Art. 48) and the EU database. Technical documentation (Art. 11 and Annex IV) carries no machine-readability requirement. The list does not re…
  Proposed correction. Replace the list with the actual provider obligations for high-risk systems: 'risk management (Art. 9), data governance (Art. 10), technical documentation (Art. 11 + Annex IV), logging capability (Art. 12), transparency to deployers (Art. 13), human oversight (Art. 14), accuracy/robustness/cybersecurity (Art. 15), QMS (Art. 17), post-market monito…
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B02-034 (docs, Medium) — docs/guides/safety_compliance.md: misleading — Confidence-weighted scoring uses the classifier's probability, so 51% vs 99% confidence differ. L183: low-confidence classifications are flagged for man…
  severity Medium
  Where. docs/guides/safety_compliance.md:70 (Turkish mirror: docs/guides/safety_compliance-tr.md:87) · type: misleading
  The document says. Confidence-weighted scoring uses the classifier's probability, so 51% vs 99% confidence differ. L183: low-confidence classifications are flagged for manual review. The Advanced example pairs Llama-Guard-3-8B with scoring: confidence_weighted and min_classifier_confidence: 0.7.
  What the code does. Llama-Guard-3-8B auto-routes to generation scoring. There each verdict gets a synthetic confidence of 1.0 or 0.0, so confidence_weighted equals safe_ratio, min_classifier_confidence has no effect, and the low-confidence warning is suppressed. The formula holds only for text-classification guards.
  Code references. forgelm/safety/_classifier.py:255-269, forgelm/safety/_score_generation.py:180-217, forgelm/safety/_gates.py:301-317, forgelm/safety/_gates.py:325-338
  Proposed correction. Add: "With a generative Llama-Guard (the default meta-llama/Llama-Guard-3-8B), confidence is synthetic (1.0/0.0): confidence_weighted equals the safe ratio and min_classifier_confidence / low-confidence flagging do not apply. These need a text-classification guard."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B02-035 (docs, Medium) — docs/guides/safety_compliance.md: misleading — ForgeLM ships the prompt library in configs/safety_prompts/ (three files listed), usable as test_prompts: "configs/safety_prompts/general_safety.jsonl".
  severity Medium
  Where. docs/guides/safety_compliance.md:107-119 (Turkish mirror: docs/guides/safety_compliance-tr.md:127-139) · type: misleading
  The document says. ForgeLM ships the prompt library in configs/safety_prompts/ (three files listed), usable as test_prompts: "configs/safety_prompts/general_safety.jsonl".
  What the code does. configs/ is a repository directory, not packaged (package data covers only forgelm.safety_prompts). It holds six files (also harmful_instructions 20, misinformation 20, privacy_pii 20). For a pip install the relative path does not exist, which triggers the silent exit-0 fail-open described above. The wheel's bundled probes (forgelm/safety_prompts/default_probes.jsonl) are not mentioned.
  Code references. pyproject.toml:257-270, configs/safety_prompts/general_safety.jsonl
  Proposed correction. "The source repository's configs/safety_prompts/ (6 files, 140 prompts — not included in the pip package)…; reference an absolute path to a copied file."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B02-036 (docs, Medium) — docs/guides/safety_compliance.md: misleading — safety_results.json details entries contain prompt and response.
  severity Medium
  Where. docs/guides/safety_compliance.md:156-164 (Turkish mirror: docs/guides/safety_compliance-tr.md:179) · type: misleading
  The document says. safety_results.json details entries contain prompt and response.
  What the code does. By default (include_eval_samples: false) prompt, response, raw_verdict, classifier_error and generation_error are removed from details. Output also includes scored_unsafe_count, unscored_count, evaluation_completed and failure_reason.
  Code references. forgelm/safety/_results.py:47-49, forgelm/safety/_results.py:96-125
  Proposed correction. Note: "prompt/response appear only with evaluation.safety.include_eval_samples: true (default false for privacy)."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B02-038 (docs, Medium) — docs/guides/safety_compliance.md: wrong detail — Tree: checkpoints/compliance/ holds compliance_report.json … deployer_instructions.md, model_integrity.json, audit_log.jsonl. L254: a complete bundle…
  severity Medium
  Where. docs/guides/safety_compliance.md:257-266 (Turkish mirror: docs/guides/safety_compliance-tr.md:287-300) · type: wrong detail
  The document says. Tree: checkpoints/compliance/ holds compliance_report.json … deployer_instructions.md, model_integrity.json, audit_log.jsonl. L254: a complete bundle is generated after every training run.
  What the code does. compliance/ holds compliance_report.json, training_manifest.yaml, data_provenance.json and data_governance_report.json, plus risk_assessment.json and annex_iv_metadata.json only when their inputs exist. deployer_instructions.md and model_integrity.json are in final_model/; audit_log.jsonl is at the output_dir root. Reproduced (runB listing). A reverted run produces no compliance/ directory (runE).
  Code references. forgelm/compliance.py:2201-2236, forgelm/trainer.py:2087, forgelm/trainer.py:1496-1506, forgelm/trainer.py:354
  Proposed correction. Show the real layout: checkpoints/audit_log.jsonl, checkpoints/compliance/{…}, checkpoints/final_model/{deployer_instructions.md, model_integrity.json}. Mark risk_assessment/annex_iv as conditional and say the bundle is written for runs that reach artefact finalisation.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B02-039 (docs, Medium) — docs/guides/safety_compliance.md: wrong detail — Each audit entry has a hash field; 'verify integrity by checking that each entry's hash matches the SHA-256 of the previous line'.
  severity Medium
  Where. docs/guides/safety_compliance.md:278-283 (Turkish mirror: docs/guides/safety_compliance-tr.md:317-322) · type: wrong detail
  The document says. Each audit entry has a hash field; 'verify integrity by checking that each entry's hash matches the SHA-256 of the previous line'.
  What the code does. Entries carry no hash field. Each entry's prev_hash is the SHA-256 of the previous raw line ('genesis' on line 1). Real lines also carry run_id and operator, and _hmac when keyed.
  Code references. forgelm/compliance.py:2561-2576, forgelm/compliance.py:452-460
  Proposed correction. Remove hash from the example and write: "each entry's prev_hash must equal the SHA-256 of the previous raw line (genesis for the first)."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B02-040 (docs, Medium) — docs/guides/safety_compliance.md: wrong detail — Pipeline flow: the model is saved after all gates pass.
  severity Medium
  Where. docs/guides/safety_compliance.md:437 (Turkish mirror: docs/guides/safety_compliance-tr.md:480-494) · type: wrong detail
  The document says. Pipeline flow: the model is saved after all gates pass.
  What the code does. The model (or staging dir) is saved before any gate runs. Gates then evaluate the on-disk model and auto-revert deletes it. With auto_revert false the failing model stays in final_model/.
  Code references. forgelm/trainer.py:1600-1622
  Proposed correction. Put "Model saved" right after Training; mark the last step "Model card + integrity + compliance artifacts".
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}