Update Sqwish Router submission - #177
Open
team-sqwish wants to merge 1 commit into
Open
Conversation
Adds Sqwish Labs' new RouterArena submission. The router jointly selects among 16 model-budget actions across nine models using a cost-aware, uncertainty-penalised ensemble and a frozen low-success diversion rule. The same policy is applied to the full and robustness splits. Public sub_10 prompt text was used only as skill anchors for targeted synthetic descendants, which account for about 4% of the training data. No RouterArena answers, model outcomes, scores or labels were used to fit or tune the router. Files: - router_inference/config/sqwish-fvcta8.json: model pool and method description - router_inference/predictions/sqwish-fvcta8.json: full prediction artifact - router_inference/predictions/sqwish-fvcta8-robustness.json: robustness decisions - universal_model_names.py: register gpt-oss-20b and xiaomi/mimo-v2.5 - model_cost/model_cost.json: list pricing for the two new pool models
Author
|
/evaluate |
team-sqwish
changed the title
Router Evaluation ResultsRouter: RouterArena Metrics
Optimality Metrics
Evaluation completed by RouterArena automated workflow |
team-sqwish
marked this pull request as ready for review
August 11, 2026 16:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sqwish Labs' new router
This PR submits the new Sqwish Router from Sqwish Labs. The artifact ID used by the config and prediction files is
sqwish-fvcta8.This submission supersedes the current Sqwish Router entry (
sqwish-router). As with the accepted Nadir replacement submission in #159, please point the existing leaderboard and manifest row atsqwish-fvcta8rather than adding a second Sqwish entry.What this router is
Sqwish Labs' new router chooses among 16 model-budget actions spanning nine models. Each request is encoded once, an ensemble estimates quality and cost for every action, and the decision rule discounts predictions where the members disagree. A small low-success fallback avoids unnecessary spend when the ensemble expects every available action to struggle.
DeepSeek V4 Flash and Gemini 3.1 Flash-Lite handle roughly 75% of requests. The other seven models supply the narrower wins.
The design follows the system described in our blog, now routing model and budget jointly with a broader training corpus and a larger predictor ensemble.
Training data
The frozen training matrix contains about 486,000 eligible model-budget outcomes across 39,000 tasks. It combines broad public and owned data across mathematics, code, science, medicine, history, language, instruction following and professional work; non-intersecting examples from public source datasets represented in RouterArena; and targeted synthetic and programmatic tasks.
No RouterArena answers, model outcomes, scores or labels were used to fit or tune the router. Public prompts from
sub_10were used only as skill anchors for targeted synthetic-task generation. The resulting descendants are new, independently answered tasks and account for about 4% of the training data; only those descendants, not thesub_10rows themselves, entered the training corpus. All RouterArena rows were held out from predictor fitting and tuning.The synthetic data is not a paraphrase set. One retained example started from this
sub_10anchor:Its descendant asks:
Both answers are True. The descendant preserves the underlying surprising chronology-comparison skill, but changes both entities and the historical interval; answering it requires a separate fact. It has low lexical overlap with the parent. Candidates like this were produced by a multistage agentic pipeline with web and code tools, then passed independent answer verification, anti-replay checks, work-and-difficulty checks and cross-model scoring.
Patch contents
The patch adds the new Sqwish Router config (
sqwish-fvcta8), full predictions and robustness predictions, plus two model registrations required by selected rows. No evaluator or scoring code changes.gpt-oss-20bxiaomi/mimo-v2.5Two small maintainer notes
model_cost.jsonlistsgemini-3.1-flash-lite-previewat $0.10/$0.40, but that was never the preview model's price. It appears to have been conflated with Gemini 2.5 Flash-Lite pricing. The new Sqwish Router uses the GAgemini-3.1-flash-liteslug, and the stale preview row does not affect this submission or the current top routers; we recommend removing it.Leaderboard row details
sqwish-fvcta8/sqwish-fvcta8-robustnesssqwish-routertosqwish-fvcta8, and update its affiliation from@namitha-sqwishto Sqwish Labs; please do not add a duplicate rowHappy to adjust the artifact naming or row-replacement details to match maintainer convention.