Paix2-router

[xufan866]

the price of minimax-m3,please look at:https://platform.minimaxi.com/docs/guides/pricing-paygo;
the price of agnes-2.0-flash,please look at:https://www.agnes-ai.com/zh-Hans/docs/agnes-20-flash;
the price of THUDM/GLM-4-9B-0414 and deepseek-ai/DeepSeek-R1-0528-Qwen3-8B,please look at:https://www.siliconflow.cn/pricing

[xufan866]

/evaluate

[xufan866]

/evaluate

[xufan866]

/evaluate

[xufan866]

/evaluate

[yl231]

Hi @xufan866 — thanks for the submission (the rename to paix-router is fine, and we also received your note about the model pool; answering both here).

Why your /evaluate CI failed. None of the four routed models — MiniMax-M3, agnes-2.0-flash, deepseek-ai/DeepSeek-R1-0528-Qwen3-8B, THUDM/GLM-4-9B-0414 — are registered in the repo. The evaluator prices every model from model_cost/model_cost.json; unknown models fall back to $0, the run's total cost comes out non-positive, and the arena scorer rejects it (Invalid cost value … must be positive), which is exactly the failure your check shows.

The actual rule: RouterArena routers may use any models — you register them yourself in your submission PR. There is no closed pool. Per the README's submission steps:

1. add each model to universal_model_names.py;
2. add its per-million-token pricing to model_cost/model_cost.json, and link the provider's official pricing page in the PR description so we can verify;
3. for a new provider, add the inference endpoint in llm_inference/model_inference.py.

Pricing policy for free / promotional models. The cost axis scores each model at its standard published per-token rate, not a promotional free tier — the scorer requires a positive cost, and the leaderboard must remain reproducible after promotions end (earlier submissions that used free-tier endpoints were priced at standard rates the same way). Concretely for your pool:

• MiniMax-M3 at $0.60 / $2.40 per M — great, add exactly that.
• The three currently-free models (agnes-2.0-flash, DeepSeek-R1-0528-Qwen3-8B, GLM-4-9B-0414): please propose a positive standard rate — the provider's list price, or a comparable paid serving rate for the same open-weight model (small 8–9B open models typically serve around $0.05–$0.10 per M). Note the arena clamps cost at $0.0044 per 1K queries, so ultra-cheap models already receive nearly the maximum cost credit — "free vs. very cheap" makes almost no scoring difference.

Once the four models are registered with verifiable pricing, re-run check_config_prediction_files.py, push, and comment /evaluate — we'll review the pricing quickly. Routing across cheap small models and paid large models is exactly the trade-off this benchmark is designed to reward, so we'd be glad to see this submission complete.

[xufan866]

/evaluate

[xufan866]

/evaluate

[xufan866]

/evaluate

[jiarong0907]

/evaluate

[jiarong0907]

Hi @xufan866, please see the following log for the error. Not sure whether you can see the entire log here: https://github.com/RouteWorks/RouterArena/actions/runs/30329751160/job/90182228793

warning: The `tool.uv.dev-dependencies` field (used in `pyproject.toml`) is deprecated and will be removed in a future release; use `dependency-groups.dev` instead
Checking router: Paix2-router
Dataset split: full
================================================================================
[1] Checking config file...
✓ Config loaded from ./router_inference/config/Paix2-router.json
✓ Found 4 models in config
✓ All models in config are valid (found in ModelNameManager)
[2] Checking prediction file...
✓ Predictions loaded from ./router_inference/predictions/Paix2-router.json
[3] Checking prediction fields against dataset...
✓ Dataset loaded: 8400 entries
  Note: Found 2427 optimality entries (excluded from size check)
✓ Prediction file has correct size
✗ Found 4 field validation errors:
  - Entry 6160 (global_index: PubMedQA_286): generated_result.generated_answer is empty but success is True
  - Entry 7300 (global_index: NarrativeQA_7854): prompt mismatch with dataset
  -   Expected: Please read the following context and answer the question based on its content.
Context:  The story...
  -   Got: Please read the following context and answer the question based on its content.
Context:  The story...
[4] Checking model cost configurations...
✓ All models have cost configurations (87 models in cost file)
================================================================================
✗ VALIDATION FAILED!
Found 4 error(s). Please fix the issues above.
================================================================================
✗ Command failed (exit code 1): uv run --active router_inference/check_config_prediction_files.py Paix2-router full --check-generated-result
Deleted branch pr-164 (was c8307b8).
→ uv run --active router_inference/check_config_prediction_files.py Paix2-router full --check-generated-result
→ git worktree remove --force /home/runner/work/RouterArena/RouterArena/base/.pr_worktrees/pr-164
→ git branch -D pr-164
Error: Process completed with exit code 1.

[xufan866]

/evaluate

[github-actions]

Router Evaluation Results

Router: Paix2-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7764
Accuracy	79.71%
Total Cost	$2.262604
Avg Cost per Query	$0.000269
Avg Cost per 1K Queries	$0.2694
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7786

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.8968
Opt.Cost (Cost Efficiency)	0.6301
Opt.Acc (Accuracy vs Optimal)	1.0000

---

Evaluation completed by RouterArena automated workflow

[xufan866]

@yl231 @jiarong0907 Thank you, I finally succeeded. I'm very moved, and my efforts have not been in vain. Could you please update the Leaderboard when you have time? Thank you.

[xufan866]

/evaluate

[github-actions]

Router Evaluation Results

Router: Paix2-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7764
Accuracy	79.71%
Total Cost	$2.262604
Avg Cost per Query	$0.000269
Avg Cost per 1K Queries	$0.2694
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7786

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.8968
Opt.Cost (Cost Efficiency)	0.6301
Opt.Acc (Accuracy vs Optimal)	1.0000

---

Evaluation completed by RouterArena automated workflow

[xufan866]

/evaluate

[github-actions]

Router Evaluation Results

Router: Paix2-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7763
Accuracy	79.69%
Total Cost	$2.262604
Avg Cost per Query	$0.000269
Avg Cost per 1K Queries	$0.2694
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7786

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.8967
Opt.Cost (Cost Efficiency)	0.6292
Opt.Acc (Accuracy vs Optimal)	1.0000

---

Evaluation completed by RouterArena automated workflow

[xufan866]

@yl231 @jiarong0907 Thank you for your help. My data has been evaluated, I'm very moved, and my efforts have not been in vain. Could you please update the Leaderboard when you have time? Thank you.