Fix Grok 4.1 Fast → Grok 4.3 mispricing on the leaderboard (#166) - #172
Merged
Conversation
xAI retired `grok-4-1-fast-reasoning` on 2026-05-15 and silently redirects it to grok-4.3 (low reasoning), which is stronger and ~5-6x pricier. Submissions that selected the retired slug via xAI/OpenRouter were served grok-4.3 but billed at the retired slug's $0.20/$0.50 price, inflating their cost-aware Arena scores. Reported and audited by @loswald (thanks!). Fix (prices the model that actually answered, not the requested alias): - model_cost.json: add `grok-4.3` and `x-ai/grok-4.3` at $1.25/$2.50. - evaluate_models.py: add `has_price()`; factor the cost lookup into `_lookup_cost_info()`. - run.py: bill by `generated_result.model_used` when it is present and priced, else the router's selected model. Redirected slugs are now billed correctly. - check_config_prediction_files.py: preflight warning for retired/redirected slugs (guardrail #1 from the issue). Precision (only provably-redirected rows change; Azure-hosted Grok 4.1 is left alone): - cross-router 217 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - vllm-sr 257 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - nadir-tumbler 3571 rows (xAI, model_used absent): backfilled model_used=x-ai/grok-4.3 (retired slug + 2026-07-04 generation date) so the same mechanism reprices it. Team to be notified. - azure-model-router (4803) and nadir-cascade-v2 (3101): provider=azure, NOT subject to the xAI redirect -> unchanged. Leaderboard (accuracy held fixed, affected calls billed at grok-4.3): - Cross-Router 76.12 -> 75.75 ($0.30 -> $0.40/1K) stays #1 - vLLM-SR 75.30 -> 74.86 ($0.30 -> $0.42/1K) #2 -> #3 - Nadir-Tumbler 75.17 -> 73.44 ($0.08 -> $0.37/1K) #4 -> #5 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #166. Thanks to @loswald for the detailed, reproducible audit.
Problem
xAI retired
grok-4-1-fast-reasoningon 2026-05-15 and silently redirects it to grok-4.3 (low reasoning) — a stronger model priced ~5–6× higher ($1.25/$2.50 vs $0.20/$0.50 per M). Routers that selected the retired slug via xAI/OpenRouter were served grok-4.3 but billed at the retired slug's old price, inflating their cost-aware Arena scores. The prediction artifacts expose it directly: those rows recordprovider=openrouter,model_used=x-ai/grok-4.3.Fix — price the model that actually answered
model_cost.json: addgrok-4.3/x-ai/grok-4.3at $1.25/$2.50.run.py/evaluate_models.py: bill bygenerated_result.model_usedwhen it is present and priced, otherwise the router's selected slug (newhas_price()helper; cost lookup factored into_lookup_cost_info()). This is guardrail [Setup] GitHub CI #3 from the issue and generalizes to any future redirect.check_config_prediction_files.py: preflight warning when a submission selects a retired/redirected slug (guardrail [Setup] Pre commit #1). Warning, not hard-fail, because Azure-hosted Grok 4.1 is a legitimate use.Precision — only provably-redirected rows change
model_usedx-ai/grok-4.3x-ai/grok-4.3x-ai/grok-4.3grok-4-1-fast-reasoningis the only grok slug used anywhere in the fleet. Nadir-Tumbler recorded nomodel_used, but it selected the retired slug and was generated 2026-07-04 (7 weeks after the redirect), so per maintainer decision its grok rows are backfilled with the resolved model; the team will be notified and invited to resubmit an optimized version.Leaderboard impact (accuracy held fixed, affected calls billed at grok-4.3)
Reproduces both the current leaderboard and @loswald's independently-computed repriced numbers. Optimality columns are left as submitted (headline Arena/Accuracy/Cost corrected). Validated: ruff/ruff-format/codespell/mypy clean;
build_site_data.pyemits 27 clean rows.