Update vLLM-SR RouterArena submission - #147
Conversation
|
/evaluate |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates the vLLM semantic router artifacts by re-pointing a subset of stored “prediction” model IDs in the robustness predictions file and refreshing the router config description.
Changes:
- Swapped the
predictionfield for multiple benchmark entries invllm-sr-robustness.jsonto different model identifiers (e.g., Gemini / DeepSeek / Grok). - Updated the human-readable
descriptionstring invllm-sr.json.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| router_inference/predictions/vllm-sr-robustness.json | Updates many per-sample prediction model identifiers used for robustness runs. |
| router_inference/config/vllm-sr.json | Adjusts the semantic-router config description text. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "global index": "MMLUPro_computer science_9086", | ||
| "prompt": "Please review the subsequent multiple-choice queries and select the answer most likely to be accurate from the provided alternatives.\n\nContext: None\n\nQuestion: To which Internet Security Protocol does Encapsulating Security Payload (ESP) pertain?\n\nOptions: \nA. Secure File Transfer Protocol\nB. Secure IP Protocol\nC. Secure Socket Layer Protocol\nD. Internet Key Exchange Protocol\nE. Transport Layer Security Protocol\nF. Secure Shell Protocol\nG. Secure Http Protocol\nH. Point-to-Point Tunneling Protocol\nI. Hypertext Transfer Protocol Secure\nJ. Network Security Protocol\n\nIndicate the correct letter choice in \\boxed{X}, where X represents the correct letter. Keep any explanation or feedback concise, limited to three sentences.", | ||
| "prediction": "grok-4-1-fast-reasoning", | ||
| "prediction": "deepseek/deepseek-v4-flash", |
| "deepseek-reasoner" | ||
| ], | ||
| "default_model": "google/gemini-3.1-flash-lite", | ||
| "description": "vLLM Semantic Router submission using deterministic semantic, structural, domain-aware, and cost-aware routing over a compact model pool." | ||
| "description": "vLLM Semantic Router using deterministic signal and projection routing." |
Router Evaluation ResultsRouter: RouterArena Metrics
Optimality Metrics
Evaluation completed by RouterArena automated workflow |
4d728ff to
4cec1c2
Compare
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Optimality Metrics
Evaluation completed by RouterArena automated workflow |
|
can you help update https://github.com/RouteWorks/RouterArena#current-leaderboard @yl231 thanks |
Transcribes the official /evaluate metrics for two merged submissions and re-ranks the README leaderboard accordingly. - vLLM-SR (#147): 72.15/73.19/$0.23 -> 75.30/77.18/$0.30, opt 16.81/25.10/89.37, robustness 67.62. Reclaims #1 (its resubmission fixed the empty token_usage rows from #135, so the previously-zeroed correct answers count again). - Hybrid Router (#150): new entry at #7 — 72.08/71.38/$0.04, opt 89.87/94.19/92.81, robustness 96.67 (@mikemao27). Also adds Hybrid Router to leaderboard_manifest.yaml so the website sync (build_site_data.py) includes it; verified the regenerated leaderboard.json and flip_labels reflect both routers. Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
Heads-up on a leaderboard correction affecting vLLM-SR (#172, closes #166). xAI retired Impact: Arena 75.30 → 74.86, cost $0.30 → $0.42/1K — rank #2 → #3. Nothing is required on your end. If you'd like to re-optimize routing now that grok-4.3's real price is visible, you're welcome to resubmit. Thanks, @Xunzhuo! |
Summary
Validation
PYTHONPATH=. python3 router_inference/check_config_prediction_files.py --check-generated-result vllm-sr fullPYTHONPATH=. python3 router_inference/check_config_prediction_files.py vllm-sr robustness