Default generative Llama-Guard scorer crashes on every supported transformers 5.x: apply_chat_template returns a BatchEncoding that is passed to generate() as input_ids (CR-A06-003) #108

Wave 0 · theme D0-7: Dependency ranges and security floors · highest severity High · 3 finding(s) · effort small

Summary

An operator enables evaluation.safety with the shipped default classifier. Training finishes, the safety pass raises AttributeError on the first probe, and the run exits 2 with an empty error. No safety verdict is produced on any supported transformers version.

Primary finding

CR-A06-003 — Default generative Llama-Guard scorer crashes on every supported transformers 5.x: apply_chat_template returns a BatchEncoding that is passed to generate() as input_ids

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. _generate_guard_verdict (forgelm/safety/_score_generation.py:143-152) calls tokenizer.apply_chat_template(conversation, return_tensors='pt', truncation=True, max_length=4096) and treats the result as a tensor (.to(), model.generate(input_ids=...), .shape[-1]). In transformers>=5.3,<6, return_dict defaults to True, so the call returns a BatchEncoding and generate() raises a bare, message-less AttributeError. AttributeError is missing from the except tuple at :162, and the orchestrator catches only RuntimeError around the generative scorer (_orchestrator.py:251-263). The error escapes run_safety_evaluation. The training run and safety-eval exit 2 with error '', no safety verdict or audit event is written, and final_model/ stays on disk. The unit tests stub apply_chat_template with a plain tensor.

Where.

• forgelm/safety/_score_generation.py:162 (except tuple lacks AttributeError)
• forgelm/safety/_score_generation.py:162 (confirmed)
• forgelm/trainer.py:1693-1709 (confirmed)
• forgelm/cli/subcommands/_safety_eval.py:266-271 (confirmed)
• forgelm/safety/_score_generation.py:151
• forgelm/safety/_score_generation.py:143-152 (confirmed)
• forgelm/safety/_orchestrator.py:251-263 (only RuntimeError caught; call at :254)
• forgelm/cli/_training.py:173-180
• pyproject.toml:53
• forgelm/trainer.py:1912-1960

Evidence.

• S-G0 (no fixture): exit 2, envelope {"success": false, "error": ""}, audit pipeline.failed {"error": ""}; traceback ends at transformers/generation/utils.py:2498 called from forgelm/safety/_score_generation.py:151. S-G1..S-G8: same, with the call log showing invalid_input_passthrough. chk_act.py: apply_chat_template returns BatchEncoding on 5.3.0, 5.13.0 and 5.18.0; pyproject.toml:53 requires transformers>=5.3.0,<6.0.0.
• r5/run-main.log (main venv, no fixture): EXIT=2, AttributeError raised from _generate_guard_verdict line 151. r5/run.log (fpy --venv latest): TypeError: SFTConfig.init() got an unexpected keyword argument 'warmup_ratio', EXIT=2.
• scratch/act_type.py: type BatchEncoding on transformers 5.3.0 (lower), 5.13.0 (main) and 5.18.0 (latest). scratch/guard_verdict.py: _generate_guard_verdict(tiny-qwen) raises AttributeError from transformers/generation/utils.py:2498 via _score_generation.py:151. End-to-end: runs/gen (FIX train_guard_generative.yaml with own output_dir and auto_revert: true) gave EXIT=2 and the envelope {"success": false, "error": ""}. The audit log holds only pipeline.initialized, training.started, evaluation.lo…

Recommended fix. Pass return_dict=False (or take ['input_ids'] from the BatchEncoding) and use the attention mask. Add AttributeError to the handled tuple, or catch the BatchEncoding shape up front. Log type(e).name when str(e) is empty. Add a test that runs the real apply_chat_template.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T05-001 — Evaluation and safety models: Generation-based guard scoring (_generate_guard_verdict) treats apply_chat_template(..., return_tensors='pt') as a tensor; transformers 5 returns BatchEncoding (outdated)

severity Now

Item. Generation-based guard scoring (_generate_guard_verdict) treats apply_chat_template(..., return_tensors='pt') as a tensor; transformers 5 returns BatchEncoding · verdict: outdated · urgency: Now

State in ForgeLM. forgelm/safety/_score_generation.py:143-152 calls tokenizer.apply_chat_template(conversation, return_tensors='pt', truncation=True, max_length=4096), then .to(model.device), then model.generate(input_ids=input_ids, ...), and slices by input_ids.shape[-1]. The per-pair except tuple at :162 catches (RuntimeError, ValueError, TypeError, IndexError, KeyError). run_safety_evaluation catches only RuntimeError around the generative scorer (forgelm/safety/_orchestrator.py:254-257). pyproject.toml:53 pins transformers>=5.3.0,<6.0.0. The unit tests stub apply_chat_template to return a tensor (tests/test_safety_advanced.py:1409, :1437).

State of the ecosystem on 2026-10-01. In transformers v5 (GA 2026-01-26), apply_chat_template returns a BatchEncoding by default; see the v5 migration guide. The arbiter reproduced the failure independently (tmp/a review run/a1-W04/repro_guard_arb.py, tiny-qwen fixture, snapshot forgelm). On transformers 5.3.0, 5.13.0 and 5.18.0, apply_chat_template returns BatchEncoding and _generate_guard_verdict raises AttributeError from transformers/generation/utils.py generate() via BatchEncoding.getattr. An unrelated project hit the same break on 2026-09-24 (Soup issue #​1238).

Impact. The shipped default safety gate (Llama-Guard-3-8B under auto) cannot score any pair on any transformers version ForgeLM allows. The AttributeError escapes the per-pair tuple and the orchestrator's RuntimeError catch. forgelm safety-eval exits 2 (cli/subcommands/_safety_eval.py:266-271), and an in-run gate fails the training pipeline. This is fail-closed: no false PASS, so threat-model hook 3 does not apply. Still, the advertised default safety gate is unusable, and the mocked tests hide the break. Every other generative guard (LG2, LG4, Qwen3Guard and so on) would hit the same line.

Recommendation. Call apply_chat_template(..., return_dict=True), pass **inputs to generate, and slice by inputs['input_ids'].shape[-1]. Alternatively, request return_dict=False explicitly. Add a non-mocked test that runs _generate_guard_verdict against a real tiny tokenizer and model (the tiny-qwen fixture is enough). Widen the per-pair fail-closed boundary to include AttributeError. Fix docs/usermanuals/*/evaluation/safety.md:37 and :203 until this ships.

Sources.

• Transformers v5 migration guide (section 4: apply_chat_template returns BatchEncoding) (living document (v5 GA 2026-01-26))
• Transformers v5.0.0 release (2026-01-26)
• Soup #1238: on transformers 5 apply_chat_template returns a BatchEncoding, and .shape raises a bare AttributeError (2026-09-24)

DOC-D3B08-025 — docs/usermanuals/en/evaluation/safety.md: false claim — The default meta-llama/Llama-Guard-3-8B works out of the box via generation-based scoring

severity High

Where. docs/usermanuals/en/evaluation/safety.md:37; docs/reference/safety_eval_subcommand.md:24,69; docs/reference/configuration.md:208 · type: false claim

The document says. The default meta-llama/Llama-Guard-3-8B works out of the box via generation-based scoring

What the code does. Generation-based scoring raises AttributeError on transformers 5.3-5.18 (apply_chat_template returns BatchEncoding); every run exits 2

Code references. forgelm/safety/_score_generation.py:143-151, forgelm/safety/_score_generation.py:143

User impact. Safety gate with the default classifier never produces a verdict

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}