Guard routing relies on five exact Hub ids and an incomplete architecture-suffix head check; safety-eval cannot override it, and ShieldGemma / Llama Guard 4 support is claimed but… (CR-A06-019) #540

Wave 2 · theme D2-8: Docs tied to medium-effort code fixes · highest severity High · 10 finding(s) · effort medium

Summary

An air-gapped operator copies Llama-Guard-3-8B to /models/llama-guard-3 and runs safety-eval. It is routed to classification, refused, and exits 2, with advice to use 'auto'. A ShieldGemma user who followed the docs gets no evaluation, and a training run exits 0 with the model kept. A ...ForConditionalGeneration guard gets a random head whose verdicts delete the model under auto_revert.

Primary finding

CR-A06-019 — Guard routing relies on five exact Hub ids and an incomplete architecture-suffix head check; safety-eval cannot override it, and ShieldGemma / Llama Guard 4 support is claimed but absent

severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. _resolve_classifier_mode('auto') (forgelm/safety/_classifier.py:255-269) selects generation only when the lowercased path is one of the five ids in _GENERATION_ONLY_CLASSIFIERS (:29-37). Local or mirrored Llama-Guard copies, Llama Guard 4 and ShieldGemma go to the text-classification pipeline, where they are refused or fail to load. safety-eval passes no classifier_mode and has no flag for one (_safety_eval.py:255-265), and its refusal message points at a key it never reads. _reject_uninitialized_classifier_head (:98) treats only *ForCausalLM / *LMHeadModel as generative, so a *ForConditionalGeneration checkpoint with a random head passes the check and every row is scored unsafe. Docs name ShieldGemma as supported.

Where.

• forgelm/safety/_classifier.py:29-37 (confirmed)
• forgelm/safety/_classifier.py:98-123 (confirmed)
• forgelm/safety/_classifier.py:113-123
• forgelm/safety/_classifier.py:255-269 (confirmed)
• docs/reference/architecture.md:134
• docs/guides/safety_compliance.md:26
• docs/reference/audit_event_catalog.md:34
• forgelm/cli/subcommands/_safety_eval.py:196-202
• forgelm/cli/subcommands/_safety_eval.py:255-265 (confirmed)
• forgelm/safety/_classifier.py:29-36

Evidence.

• scratch/mappings.py: seq-cls mapping llama4=None, gemma2=Gemma2ForSequenceClassification, gemma3=Gemma3ForSequenceClassification. fixtures/condgen-guard is tiny-qwen with architectures renamed to Qwen2ForConditionalGeneration. safety-eval on it gave EXIT=3 with 10/10 rows LABEL_1 (confidence 0.5004), scored_unsafe 10 and evaluation_completed True, while transformers printed 'MISSING ... newly initialized'. Copy: outputs/.../runs/condgen.log.
• runs/se_localgen (--classifier tiny-qwen local causal LM, no scenario): exit 2, failure_reason as quoted. safety-eval --help lists no classifier-mode option.
• run5: classifier = local causal LM, mode auto, gives audit.classifier_load_failed ('is a causal language model ... loaded as a text-classification head'), then EXIT=0 with the model kept (auto_revert true). Code read: _safety_eval.py:255-265 passes no classifier_mode. The ShieldGemma and Llama Guard 4 outcomes were inferred from the architecture-suffix check and the verdict grammar; they were not run (needs real weights).

Recommended fix. Detect generative guards from the loaded config (model_type, architectures, chat template) rather than repo ids. Treat any architecture outside the seq-cls family as causal, or check output_loading_info for a newly initialised score head. Expose classifier_mode in safety-eval. Remove or qualify the ShieldGemma claims and list Llama Guard 4 as unsupported.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T05-002 (technology, Next release) — Evaluation and safety models: Default safety classifier meta-llama/Llama-Guard-3-8B and the generative allow-list (llama-guard-3-8b, -3-8b-int8, -3-1b, meta-llama-guard-2-8b, llamaguard-7b); no Llama…
  severity Next release
  Item. Default safety classifier meta-llama/Llama-Guard-3-8B and the generative allow-list (llama-guard-3-8b, -3-8b-int8, -3-1b, meta-llama-guard-2-8b, llamaguard-7b); no Llama Guard 4 path · verdict: aging · urgency: Next release
  State in ForgeLM. forgelm/config.py:914-915 sets the classifier default to meta-llama/Llama-Guard-3-8B. forgelm/safety/_classifier.py:29-37 holds the five-id allow-list, compared case-insensitively. _resolve_classifier_mode (_classifier.py:255-269) sends allow-listed ids to generation scoring and every other id to the text-classification pipeline under classifier_mode=auto. _load_generative_guard (_classifier.py:2…
  State of the ecosystem on 2026-10-01. Llama-Guard-3-8B is still on the Hub, gated (manual), LlamaForCausalLM, created 2024-07-22 and last modified 2024-10-11. Meta's current guard is Llama-Guard-4-12B: created 2025-04-23, Llama4ForConditionalGeneration, pipeline image-text-to-text, gated. Its card says it 'combines the capabilities of the previous Llama Guard 3-8B and Llama Guard 3-11B-vision models' and 'roughly matches or exceeds'…
  Recommendation. Keep LG3-8B as the default for now. After fixing the apply_chat_template defect, add first-class LG4 support: an allow-list entry, AutoProcessor + Llama4ForConditionalGeneration loading, and typed-content messages. Validate it with real weights. Extend _reject_uninitialized_classifier_head to cover *ForConditionalGeneration. Move to per-family verdict parsers instead of allow-listing by id.
  Sources.

  • HF Hub API: meta-llama/Llama-Guard-4-12B (2025-04-29 (lastModified))
  • HF Hub API: meta-llama/Llama-Guard-3-8B (2024-10-11 (lastModified))

• TECH-T05-007 (technology, Next release) — Evaluation and safety models: ShieldGemma named in docs as a supported safety classifier (incorrect-claim)
  severity Next release
  Item. ShieldGemma named in docs as a supported safety classifier · verdict: incorrect-claim · urgency: Next release
  State in ForgeLM. The docs name ShieldGemma at docs/guides/safety_compliance.md:26 ('Llama Guard, ShieldGemma, or custom'; TR safety_compliance-tr.md:37), docs/reference/architecture.md:134, docs/reference/audit_event_catalog.md:34 (TR :34) and the completed-phase record docs/roadmap/completed-phases.md:111. rg over forgelm/ finds no ShieldGemma identifier, prompt builder or Yes/No scorer, and no ShieldGemma id is…
  State of the ecosystem on 2026-10-01. ShieldGemma 2b/9b/27b (HF createdAt 2024-07-16) is Gemma2ForCausalLM. It answers Yes/No against one supplied guideline and is designed to run in scoring mode on the Yes/No next-token probability. Its HF chat template references a guideline variable. ShieldGemma 2 (google/shieldgemma-2-4b-it, 2025-03-04) is ShieldGemma2ForImageClassification, an image-only safety classifier.
  Recommendation. Remove ShieldGemma from the EN/TR guides, architecture.md and the audit catalog. Alternatively, implement a ShieldGemma-1 backend: a policy prompt plus P(Yes) logit scoring over its four harm types. Never present ShieldGemma 2 as a text guard. Consider failing fast on google/shieldgemma-* ids with an actionable error.
  Sources.

  • ShieldGemma model card (2025-02-25 (last updated))
  • ShieldGemma 2 model card (2025-04-03 (last updated))

• TECH-T14-007 (technology, Watch) — Hard-coded model identifiers and model-merging methods: Safety classifier default meta-llama/Llama-Guard-3-8B and the generation-only allowlist (llama-guard-3-8b-int8, llama-guard-3-1b, meta-llama-gu…
  severity Watch
  Item. Safety classifier default meta-llama/Llama-Guard-3-8B and the generation-only allowlist (llama-guard-3-8b-int8, llama-guard-3-1b, meta-llama-guard-2-8b, llamaguard-7b) (forgelm/config.py:915, forgelm/safety/_classifier.py:29-37) · verdict: aging · urgency: Watch
  State in ForgeLM. config.py:915 defaults evaluation.safety.classifier to meta-llama/Llama-Guard-3-8B; the same default appears in config_template.yaml:194 (commented), cli/subcommands/_safety_eval.py:202 and wizard/_collectors.py:309. safety/_classifier.py:29-37 lists five generative Llama-Guard ids that are routed to generation scoring. _parse_guard_verdict (safety/_score_generation.py:46-79) accepts a first line…
  State of the ecosystem on 2026-10-01. All five allowlisted ids exist, each gated=manual under a Llama licence: Guard-3-8B and -8B-INT8 llama3.1, Guard-3-1B llama3.2, Guard-2-8B llama3, LlamaGuard-7b llama2. Meta's newer guard, Llama-Guard-4-12B (2025-04), is gated=manual under the llama4 licence and is a multimodal Llama4ForConditionalGeneration (image-text-to-text). The Llama 4 AUP excludes EU-domiciled licensees from multimodal Lla…
  Recommendation. Keep Llama-Guard-3-8B (text-only, Llama 3.1 licence, no EU clause in the 3.1 AUP). Do not migrate to Llama-Guard-4 without flagging the EU AUP restriction. Before documenting an ungated Apache-2.0 guard as an option, add a per-guard verdict parser (e.g. for Qwen3Guard's 'Safety:' format).
  Sources.

  • HF API: meta-llama/Llama-Guard-3-8B (gated=manual, llama3.1, LlamaForCausalLM) (2024-10-11 (lastModified))
  • HF API: meta-llama/Llama-Guard-4-12B (gated=manual, licence other/llama4, Llama4ForConditionalGeneration, image-text-to… (2025-04-29 (lastModified))

• DOC-D2B02-010 (docs, High) — docs/guides/safety_compliance.md: false claim — The safety classifier can be Llama Guard, ShieldGemma, or custom.
  severity High
  Where. docs/guides/safety_compliance.md:26 (Turkish mirror: docs/guides/safety_compliance-tr.md:37) · type: false claim
  The document says. The safety classifier can be Llama Guard, ShieldGemma, or custom.
  What the code does. There is no ShieldGemma support. auto mode routes unknown ids to text-classification, which refuses a causal LM with a placeholder head. Generation mode parses only a Llama-Guard 'safe'/'unsafe' first line, so ShieldGemma's Yes/No output is malformed or unscored and the evaluation is reported as not performed. 'shieldgemma' appears nowhere in forgelm/ or tests/.
  Code references. forgelm/safety/_classifier.py:29-37, forgelm/safety/_classifier.py:255-269, forgelm/safety/_classifier.py:70-120, forgelm/safety/_score_generation.py:46-75
  Proposed correction. "…A safety classifier (a Llama-Guard checkpoint via generation scoring, or a custom text-classification model whose labels are safe/unsafe)…" — remove ShieldGemma.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B02-011 (docs, High) — docs/guides/safety_compliance.md: misleading — 'If the unsafe ratio exceeds your threshold, ForgeLM auto-reverts the model', and the Fail-Safe list states 'Exit code 3 is returned' unconditionally (L…
  severity High · already tracked: KNOWN-OPEN DOCS-02/14/15/16 (S15 unconditional exit 3) — docs/roadmap/phase-16-trust-surface-hardening.md:776-777
  Where. docs/guides/safety_compliance.md:27 (Turkish mirror: docs/guides/safety_compliance-tr.md:37-40) · type: misleading
  The document says. 'If the unsafe ratio exceeds your threshold, ForgeLM auto-reverts the model', and the Fail-Safe list states 'Exit code 3 is returned' unconditionally (L185).
  What the code does. auto_revert defaults to false. Then a failed safety (or benchmark/judge) gate is only logged: the model is kept and the run exits 0. This is documented in the error-handling standard. Reproduced (B): 100% unsafe with auto_revert false gave EXIT=0 and pipeline.completed success=True.
  Code references. forgelm/config.py:1191-1194, forgelm/trainer.py:1460-1462, docs/standards/error-handling.md:22
  Proposed correction. "3. If the unsafe ratio exceeds your threshold and evaluation.auto_revert: true, ForgeLM deletes the model and exits 3. With the default auto_revert: false the failure is only recorded (safety.passed: false) and the run exits 0."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B07-002 (docs, High) — docs/roadmap/completed-phases.md: false claim — [x] shipped: 'Compare safety scores before vs after fine-tuning' / 'Before/After Safety Comparison' that runs safety eval on the base model and the fin…
  severity High
  Where. docs/roadmap/completed-phases.md:111, 537-552 · type: false claim
  The document says. [x] shipped: 'Compare safety scores before vs after fine-tuning' / 'Before/After Safety Comparison' that runs safety eval on the base model and the fine-tuned model and reports base_model_safety_score, regression, new_unsafe_prompts and newly_safe_prompts.
  What the code does. No pre-training safety measurement exists anywhere. The gate is an absolute unsafe-ratio ceiling on the post-training model only, and no comparison report is emitted.
  Code references. forgelm/safety/__init__.py:7-11, forgelm/safety/_gates.py:255-261, forgelm/safety/_results.py:108-129
  Proposed correction. Untick Phase 6 task 1's 'compare before vs after' clause and Phase 9 task 6, or mark them '[ ] not implemented: the safety gate is an absolute unsafe-ratio ceiling on the fine-tuned model; no base-model run or comparison report exists'.

• DOC-D1B01-015 (docs, Medium) — docs/reference/architecture.md: false claim — "Runs a configurable safety classifier (Llama Guard, ShieldGemma)"
  severity Medium
  Where. docs/reference/architecture.md:134 (Turkish mirror: docs/reference/architecture-tr.md:133 (no ShieldGemma mention)) · type: false claim
  The document says. "Runs a configurable safety classifier (Llama Guard, ShieldGemma)"
  What the code does. Only the Llama-Guard protocol is supported. The generation scorer accepts a first line of 'safe' or 'unsafe' plus S-codes. The classification path decides by whether the label contains 'safe'/'unsafe'. ShieldGemma (Yes/No output) is not handled anywhere: its verdicts would be malformed, i.e. unscored, which ends in evaluation_completed=False.
  Code references. forgelm/safety/_score_generation.py:46-77, forgelm/safety/_score_classification.py:45-48, forgelm/safety/_classifier.py:29-35, forgelm/safety/_gates.py:39-60
  Proposed correction. "Runs a Llama-Guard-protocol safety classifier (generative Llama Guard checkpoints, or a text-classification model whose labels are safe/unsafe)."
  Turkish mirror. the Turkish page has no such statement

• DOC-BLEGAL-250 (docs, Low) — notebooks/safety_evaluation.ipynb: wrong detail — Accept the Meta Llama 3 Community License
  severity Low
  Where. notebooks/safety_evaluation.ipynb cell 3 · type: wrong detail
  The document says. Accept the Meta Llama 3 Community License
  What the code does. The meta-llama/Llama-Guard-3-8B repository carries license 'llama3.1', and its gate shows the 'LLAMA 3.1 COMMUNITY LICENSE AGREEMENT' (release date July 23, 2024). Llama 3 and Llama 3.1 are separate agreements. The '~16 GB in FP16' figure in the same cell is correct (RQ-542).
  Proposed correction. 'Accept the Llama 3.1 Community License'.

• DOC-D2B02-067 (docs, Low) — docs/guides/safety_compliance.md: wrong detail — The classifier evaluates prompt+response 'using the Llama Guard 3 format: [INST] {prompt} [/INST] {response}'.
  severity Low
  Where. docs/guides/safety_compliance.md:21 (Turkish mirror: docs/guides/safety_compliance-tr.md:29) · type: wrong detail
  The document says. The classifier evaluates prompt+response 'using the Llama Guard 3 format: [INST] {prompt} [/INST] {response}'.
  What the code does. That string is used only on the text-classification path. Llama-Guard-3 checkpoints, including the default, run in generation mode through the tokenizer's chat template (user = probe, assistant = response).
  Code references. forgelm/safety/_score_classification.py:40, forgelm/safety/_score_generation.py:139-147
  Proposed correction. "…evaluates the full conversation: generative Llama-Guard checkpoints via their chat template; text-classification guards via [INST] {prompt} [/INST] {response}."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}