Evaluation and safety models: LLM judge default model gpt-4o (and gpt-4o-mini in the wizard/docs) (aging) (TECH-T05-008) #490

Wave 2 · theme D2-7: Next-release technology updates · highest severity Medium · 3 finding(s) · effort not estimated

Summary

The default works today. It is a previous-generation 2024 model, and judge scores are not comparable across judge models. The 'gpt-4' teacher examples (config_template.yaml:105, forgelm/config.py:1331, docs/usermanuals/en/reference/configuration.md:176) name a model that shuts down on 2026-10-23.

Primary finding

TECH-T05-008 — Evaluation and safety models: LLM judge default model gpt-4o (and gpt-4o-mini in the wizard/docs) (aging)

severity Next release

Item. LLM judge default model gpt-4o (and gpt-4o-mini in the wizard/docs) · verdict: aging · urgency: Next release

State in ForgeLM. JudgeConfig.judge_model defaults to 'gpt-4o' (forgelm/config.py:1086-1087). _call_api_judge defaults model='gpt-4o' and posts to https://api.openai.com/v1/chat/completions (forgelm/judge.py:82, :223, :233). The wizard defaults the judge to gpt-4o-mini (forgelm/wizard/_collectors.py:979) and the teacher to gpt-4o (:1020). docs/usermanuals/en/evaluation/judge.md:69-70 lists gpt-4o as 'Default for production' and gpt-4o-mini as the cost-balanced default.

State of the ecosystem on 2026-10-01. The OpenAI gpt-4o page lists the gpt-4o alias, gpt-4o-2024-08-06, gpt-4o-2024-11-20 and gpt-4o-2024-05-13 (marked Deprecated), and lists the Chat Completions endpoint. The deprecations page (newest entry 2026-09-11) shuts down gpt-4o-2024-05-13 on 2026-10-23 (substitute gpt-5.6-sol). The same 2026-04-22 notice shuts down gpt-4-0613|gpt-4. Neither the gpt-4o alias nor gpt-4o-mini is scheduled. The models page tells developers to start with GPT-6 Astra (gpt-6-astra), GPT-6.1 Sol (gpt-6.1-sol) or GPT-6 Luna (gpt-6-luna). Deprecation tables name gpt-5.6-sol/terra/luna as replacements.

Impact. The default works today. It is a previous-generation 2024 model, and judge scores are not comparable across judge models. The 'gpt-4' teacher examples (config_template.yaml:105, forgelm/config.py:1331, docs/usermanuals/en/reference/configuration.md:176) name a model that shuts down on 2026-10-23.

Recommendation. Keep gpt-4o, pinned to a dated snapshot for reproducibility, until the payload is made reasoning-model-safe (separate item). Then consider a current mid-tier default. Replace the 'gpt-4' examples before 2026-10-23. Record the exact judge snapshot in the audit trail.

Sources.

• OpenAI API Deprecations (live page; newest entry 2026-09-11)
• GPT-4o Model | OpenAI API (live page)
• OpenAI Models overview (live page)

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

DOC-D4B02-007 — config_template.yaml: broken example — Synthetic block example with teacher_backend: "api" and teacher_revision: "6f6073b4..." on adjacent lines, presented as a block to uncomment

severity Medium

Where. config_template.yaml:105-107 · type: broken example

The document says. Synthetic block example with teacher_backend: "api" and teacher_revision: "6f6073b4..." on adjacent lines, presented as a block to uncomment

What the code does. Uncommented verbatim, the block fails validation: "synthetic.teacher_revision pins a locally-loaded teacher model, but teacher_backend='api' never loads teacher_model from the Hub" (ValidationError, exit 1). The inline comment says "local backend only", but the example combines the two anyway.

Code references. forgelm/config.py:1401-1418

User impact. Copy-pasting the documented synthetic block gives a config error.

Proposed correction. Comment out the teacher_revision line inside the block: "# # teacher_revision: "<40-hex sha>" # ONLY with teacher_backend: "local" — rejected (exit 1) with "api"/"file""

DOC-BFIELD-032 — forgelm/config.py: misleading — api (OpenAI/Anthropic)

severity Low

Where. forgelm/config.py:1333-1336 (SyntheticConfig.teacher_backend, F38) · type: misleading

The document says. api (OpenAI/Anthropic)

What the code does. The api backend POSTs an OpenAI chat/completions body with a Bearer header to {api_base}/chat/completions and reads choices[0].message.content (synthetic.py:218-247). The native Anthropic Messages API is not supported; only an OpenAI-compatible endpoint works. The class docstring (synthetic.py:47) itself says 'OpenAI-compatible API'.

Code references. forgelm/synthetic.py:208-247

Proposed correction. 'api (any OpenAI-compatible chat/completions endpoint)'.

Acceptance criteria

• Each statement is re-checked against the code at fix time.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}