llm_judge.judge_api_base is posted to verbatim as the full endpoint, but is described and exemplified as a base URL (CR-A06-008) #123

Wave 1 · theme D1-1: Correctness of the evaluation gates · highest severity High · 5 finding(s) · effort medium

Summary

judge_api_base: https://api.openai.com/v1 as in safety_compliance.md. Each request is POST https://api.openai.com/v1, the provider returns an HTTP error, and the gate fails on every run.

Primary finding

CR-A06-008 — llm_judge.judge_api_base is posted to verbatim as the full endpoint, but is described and exemplified as a base URL

severity High · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)

What happens. _call_api_judge posts to api_base or OPENAI_API_BASE (forgelm/judge.py:233), and the default constant is the full .../v1/chat/completions URL (judge.py:82). The field description (config.py:1092-1094) and the doc examples (configuration.md:228, safety_compliance.md:205, judge.md:71) give bases such as https://api.openai.com/v1, while synthetic.api_base appends /chat/completions (synthetic.py:241). A documented value POSTs to .../v1, every prompt fails as a non-auth HTTP error (score None), and the gate fails with 'No valid judge scores', after which auto_revert deletes the model.

Where.

• forgelm/judge.py:82
• forgelm/judge.py:260-270
• forgelm/config.py:1092-1094
• forgelm/synthetic.py:241
• docs/reference/configuration.md:228
• docs/guides/safety_compliance.md:205
• docs/usermanuals/en/evaluation/judge.md:71
• forgelm/judge.py:248
• forgelm/judge.py:810-813
• forgelm/synthetic.py:218

Evidence.

• runs/judge_base (judge-api-valid, judge_api_base https://judge.scripted.forgelm.test/v1): calls.jsonl served request {"method": "POST", "url": "/v1"}. The fixture answers any path, so this run exited 0. The provider-side 404 is not reproduced offline.
• judge_url.py with safe_post stubbed recorded the URLs posted: judge with api_base 'https://api.openai.com/v1' posted to 'https://api.openai.com/v1'; synthetic with the same api_base posted to '.../v1/chat/completions'. safe_post refused http://127.0.0.1:11434, https://127.0.0.1:8000 and https://10.0.0.5 (HttpSafetyError).
• judge_probe.py: api_base='https://api.openai.com/v1' -> POST https://93.184.216.34/v1 (Host api.openai.com) -> {'score': None, 'reason': 'API error: 404 ...'}; api_base=None -> POST .../v1/chat/completions. Existing tests pass full paths ('https://custom.api/v1/chat', tests/test_judge_functions.py:288), so the mismatch is untested.

Recommended fix. Treat judge_api_base as a base and append /chat/completions when the path does not already end with it (matching synthetic.py), or rename and redocument it as a full endpoint URL and fix the three examples.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• CR-A06-013 (code, High) — Anthropic/Claude advertised as an API judge, but only the OpenAI chat-completions wire format is implemented
  severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs
  Failure scenario. An operator configures a Claude judge the way judge.md:71 shows. Every judge call fails and the judge gate never yields a score.
  Where. forgelm/judge.py:223-284 (_call_api_judge; url at :233, Bearer header at :235, OpenAI payload at :238-243, choices parse at :250), docs/guides/safety_compliance.md:197, docs/usermanuals/en/evaluation/judge.md:71, forgelm/judge.py:82
  Recommended fix. Document the exact working Anthropic configuration (the full OpenAI-compatible endpoint URL, if it exists) or drop the Anthropic claim; alternatively add a native Messages-API adapter.

• TECH-T05-005 (technology, Next release) — Evaluation and safety models: Judge docs: Anthropic judge example (claude-haiku-4 + judge_api_base https://api.anthropic.com/v1) and judge_api_base base-URL vs endpoint semantics (incorrect-claim)
  severity Next release
  Item. Judge docs: Anthropic judge example (claude-haiku-4 + judge_api_base https://api.anthropic.com/v1) and judge_api_base base-URL vs endpoint semantics · verdict: incorrect-claim · urgency: Next release
  State in ForgeLM. docs/usermanuals/en/evaluation/judge.md:71 (TR :71) recommends claude-haiku-4 with judge_api_base: https://api.anthropic.com/v1. forgelm/judge.py:233 posts to url = api_base or OPENAI_API_BASE verbatim and does not append /chat/completions. The unit test passes a full endpoint (tests/test_judge_functions.py:476, 'https://custom.llm.api/v1/chat/completions'). By contrast, synthetic.py:241 appe…
  State of the ecosystem on 2026-10-01. Anthropic's OpenAI-SDK compatibility layer uses base_url https://api.anthropic.com/v1/, and the SDK appends /chat/completions. The Claude models overview lists claude-haiku-4-5 and claude-haiku-4-5-20251001. 'claude-haiku-4' is not a listed id, neither current nor legacy.
  Recommendation. Pick one semantics. Either append /chat/completions in judge.py when given a base URL, as synthetic.py does, or document judge_api_base as the full endpoint everywhere (EN and TR). Replace claude-haiku-4 with a real id such as claude-haiku-4-5.
  Sources.

  • OpenAI SDK compatibility (Claude API docs) (live page)
  • Claude Models overview (live page)

• DOC-D3B08-024 (docs, High) — docs/usermanuals/en/evaluation/judge.md: broken example — An Anthropic judge works by setting judge_api_base: https://api.anthropic.com/v1 plus the right env var.
  severity High
  Where. docs/usermanuals/en/evaluation/judge.md:71 (Turkish mirror: docs/usermanuals/tr/evaluation/judge.md:71) · type: broken example
  The document says. An Anthropic judge works by setting judge_api_base: https://api.anthropic.com/v1 plus the right env var.
  What the code does. judge_api_base is used verbatim as the POST URL (url = api_base or OPENAI_API_BASE, where the default is the full .../v1/chat/completions endpoint), with an OpenAI chat-completions payload and a Bearer header (judge.py:82, 233-248). POSTing to https://api.anthropic.com/v1 is not an endpoint, so every prompt scores None and the gate fails with 'No valid judge scores' (judge.py:810-815).
  Code references. forgelm/judge.py:82, forgelm/judge.py:223-251, forgelm/judge.py:810-815
  Proposed correction. State that judge_api_base must be the full OpenAI-compatible chat-completions URL (e.g. https://<host>/v1/chat/completions), and only list a provider row once its OpenAI-compatible endpoint, auth scheme and model id are verified.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BFIELD-016 (docs, Medium) — forgelm/config.py: wrong detail — Override the judge API base URL (Azure OpenAI, self-hosted vLLM, etc.).
  severity Medium
  Where. forgelm/config.py:1092-1094 (JudgeConfig.judge_api_base, F04) (Turkish mirror: docs/reference/configuration.md:228 (example https://api.together.xyz/v1 would POST to /v1); docs/usermanuals/tr/evaluation/judge.md:71 (https://api.anthropic.com/v1)) · type: wrong detail
  The document says. Override the judge API base URL (Azure OpenAI, self-hosted vLLM, etc.).
  What the code does. The value is the full POST URL, not a base. judge.py:233 sets url = api_base or OPENAI_API_BASE, the default is 'https://api.openai.com/v1/chat/completions' (judge.py:82), and it is posted verbatim (judge.py:248). synthetic.api_base differs: it appends /chat/completions (synthetic.py:241). Reproduced: api_base='https://vllm.example.com/v1' was POSTed to exactly that URL; a 404 gave score None,…
  Code references. forgelm/judge.py:82, forgelm/judge.py:223-248, forgelm/synthetic.py:218-245, forgelm/_http.py:528-571
  Proposed correction. Append /chat/completions when the value has no path ending in /chat/completions, as synthetic does. Or document that it is the full chat-completions URL, which must be public HTTPS.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}