_release_model_from_gpu moves the trainer's live model to CPU and nothing restores it, so the later judge gate generates on CPU (or fails for dispatched/4-bit models) (CR-A02-019) #119

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 1 finding(s) · effort medium

Summary

QLoRA 7B run on CUDA with safety and llm_judge enabled. After the safety pass the model is on CPU, and 200 judge prompts x 512 new tokens run on CPU for hours. With a 4-bit model, CPU generation may raise, every response becomes '' and is judged as valid.

Primary finding

CR-A02-019 — _release_model_from_gpu moves the trainer's live model to CPU and nothing restores it, so the later judge gate generates on CPU (or fails for dispatched/4-bit models)

severity High · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · not reproduced (code trace)

What happens. run_safety_evaluation calls _release_model_from_gpu(model) (forgelm/safety/_orchestrator.py:239-244), which runs model.cpu() (forgelm/safety/_generate.py:157) on the object self.trainer still holds; clearing the local reference frees nothing. Gate order is benchmark, safety, judge (forgelm/trainer.py:1641-1650), and _run_judge_if_configured passes the same self.trainer.model (trainer.py:1986-1988) to the judge, which places inputs on model.device (forgelm/judge.py:290, 390, 427). No code moves the model back to the accelerator.

Where.

• forgelm/trainer.py:1646 and 1649 (gate order)
• forgelm/safety/_orchestrator.py:240 (call), :244 (local-only reference drop)
• forgelm/safety/_generate.py:143-184 (helper; model.cpu() at :157)
• forgelm/judge.py:390, forgelm/judge.py:427 (inputs placed on model.device)
• forgelm/judge.py:427 (confirmed)
• forgelm/safety/_generate.py:143-184 (confirmed)
• forgelm/safety/_orchestrator.py:239-244 (confirmed)
• forgelm/trainer.py:1646-1649 (confirmed)
• forgelm/trainer.py:1987 (judge receives the same self.trainer.model)
• forgelm/judge.py:290, 390, 427 (model.device); swallow at judge.py:394-401

Evidence.

• Static only. forgelm/safety/_orchestrator.py:240 calls _release_model_from_gpu. forgelm/safety/_generate.py:157 runs model.cpu(). No .to/.cuda on self.trainer.model in trainer.py after the safety stage (grep of trainer.py finds .to( only in the reward-model path at 1181). The fixture host has no CUDA, so this was not reproduced.
• Code reading. scratch/cpu_override.py: PreTrainedModel.cpu is nn.Module.cpu, while .to and .cuda are overridden; PeftModel.cpu is nn.Module.cpu; bitsandbytes is not importable. runs/devspy could not show a device change because the trainer model was already on CPU on this host.
• rg finds no .to(device)/.cuda() restore anywhere in forgelm. Run (MPS standing in for CUDA): model on mps:0 -> _release_model_from_gpu -> model.device == cpu -> forgelm.judge._generate_responses_batched runs with inputs on cpu. Multi-device case: tiny-qwen dispatched across mps+cpu with accelerate dispatch_model, generation works before release; after release the judge-stage call logs 'Placeholder storage has not been allocated on MPS device!' and returns ['', ''] (plumbing-only analogue: my cp…

Recommended fix. Record the model's device before releasing it and restore it after the safety pass (or release only the guard), or run the judge before the safety gate.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}