Llama Guard 2 and Llama Guard 1 are on the generation allow-list but are scored against the Llama Guard 3 S1-S14 taxonomy, so categories and severities are wrong (CR-A06-077) #536

Wave 2 · theme D2-7: Next-release technology updates · highest severity Info · 2 finding(s) · effort small

Summary

_GENERATION_ONLY_CLASSIFIERS lists meta-llama/meta-llama-guard-2-8b and meta-llama/llamaguard-7b as supported generative guards (_classifier.py:34-35), but category and severity mapping always uses Llama Guard 3's taxonomy (HARM_CATEGORIES / CATEGORY_SEVERITY, _types.py:17-50). Llama Guard 2 uses a different S1-S11 numbering: S5 Specialized Advice, S8 Indiscriminate Weapons, S9 Hate, S10 Suicide & Self-Harm, S11 Sexual Content (PurpleLlama Llama-Guard2/MODEL_CARD.md). So an LG2 'unsafe\nS8' (weapons) is reported as intellectual_property with severity 'low', and S10 (self-harm) as hate/'high'. A severity_thresholds gate such as {'critical': 0.0} under-counts critical harms. Llama Guard 1 uses six O-coded categories with no S-codes, so no category or severity is ever recorded and the severity gate is inert. The binary unsafe-ratio gate is unaffected. Currently latent: H-A06-01 blocks the whole generation path.

Primary finding

CR-A06-077 — Llama Guard 2 and Llama Guard 1 are on the generation allow-list but are scored against the Llama Guard 3 S1-S14 taxonomy, so categories and severities are wrong

severity Info · reachability undocumented · intent contract-violation · confidence likely · single reviewer · not reproduced (code trace)

What happens. _GENERATION_ONLY_CLASSIFIERS lists meta-llama/meta-llama-guard-2-8b and meta-llama/llamaguard-7b as supported generative guards (_classifier.py:34-35), but category and severity mapping always uses Llama Guard 3's taxonomy (HARM_CATEGORIES / CATEGORY_SEVERITY, _types.py:17-50). Llama Guard 2 uses a different S1-S11 numbering: S5 Specialized Advice, S8 Indiscriminate Weapons, S9 Hate, S10 Suicide & Self-Harm, S11 Sexual Content (PurpleLlama Llama-Guard2/MODEL_CARD.md). So an LG2 'unsafe\nS8' (weapons) is reported as intellectual_property with severity 'low', and S10 (self-harm) as hate/'high'. A severity_thresholds gate such as {'critical': 0.0} under-counts critical harms. Llama Guard 1 uses six O-coded categories with no S-codes, so no category or severity is ever recorded and the severity gate is inert. The binary unsafe-ratio gate is unaffected. Currently latent: H-A06-01 blocks the whole generation path.

Where.

• forgelm/safety/_classifier.py:34
• forgelm/safety/_classifier.py:35
• forgelm/safety/_types.py:17
• forgelm/safety/_types.py:35
• forgelm/safety/_score_generation.py:235

Evidence.

• Static: the taxonomy table is Llama Guard 3's. The LG2 taxonomy comes from the PurpleLlama GitHub model card (S8 Indiscriminate Weapons, S10 Suicide & Self-Harm).

Recommended fix. Either drop LG2 and LG1 from the allow-list, or key the taxonomy by guard version and refuse severity gating for guards whose taxonomy is unknown.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T05-006 — Evaluation and safety models: Allow-listed legacy guards (Meta-Llama-Guard-2-8B, LlamaGuard-7b) scored with the Llama Guard 3 S1-S14 category/severity table (at-risk)

severity Next release

Item. Allow-listed legacy guards (Meta-Llama-Guard-2-8B, LlamaGuard-7b) scored with the Llama Guard 3 S1-S14 category/severity table · verdict: at-risk · urgency: Next release

State in ForgeLM. _classifier.py:34-35 allow-lists meta-llama/meta-llama-guard-2-8b and meta-llama/llamaguard-7b for generation scoring. Every S-code is mapped through HARM_CATEGORIES/CATEGORY_SEVERITY (forgelm/safety/_types.py:16-49), which is the LG3 table. _classify_one_generative (_score_generation.py:230-242) records category and severity from it. The severity gate counts severity_dist per tier (forgelm/safety/_gates.py:269-277).

State of the ecosystem on 2026-10-01. The Llama Guard 2 card defines S1-S11 with different meanings: S5 Specialized Advice, S6 Privacy, S7 Intellectual Property, S8 Indiscriminate Weapons, S9 Hate, S10 Suicide & Self-Harm, S11 Sexual Content. The Llama Guard (7B) card uses six named categories (Violence & Hate, Sexual Content, Guns & Illegal Weapons, Regulated Substances, Suicide & Self Harm, Criminal Planning), not S-codes. Reading ForgeLM's table: LG2 'unsafe S8' (weapons) becomes intellectual_property/low, S10 (self-harm) becomes hate/high, S5 (specialized advice) becomes defamation/medium, and S11 (sexual content) becomes suicide_self_harm/critical.

Impact. Take an allow-listed LG2 guard with track_categories and severity_thresholds set stricter than max_safety_regression, for example {critical: 0}. Weapons verdicts are then filed as 'low', so the per-severity gate under-counts critical harm. That is a partial fail-open of the severity gate, which threat-model hook 3 treats as gate-relevant. The unsafe-ratio gate still counts those verdicts as unsafe. The category reports in safety_results.json are mislabelled. LlamaGuard-7b verdicts get no S-code, so no category or severity is recorded. All of this is latent today because generation scoring currently crashes (previous item). It becomes live once that is fixed.

Recommendation. Make the category/severity table per guard family (LG2 S1-S11; LG3/LG4 S1-S14). Alternatively, drop LG2 and LlamaGuard-7b from the allow-list, or reject track_categories/severity_thresholds for them. Ship the change together with the apply_chat_template fix.

Sources.

• Meta Llama Guard 2 Model Card (not stated in file (model 2024-04))
• Llama Guard (7B) Model Card (not stated in file)
• Llama Guard 3 8B Model Card (not stated in file (model 2024-07))

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}