You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Benchmark runs against typesafe/jev-router default to adaptive reasoning effort, matching openrouter/jev.
What changed?
Added typesafe/jev-router to ADAPTIVE_REASONING_EFFORT_MODELS.
Added tests for the alias default and an explicit --reasoning-effort auto.
Why?
openrouter-web is adding typesafe/jev-router as an alias for the Jev router (OpenRouterTeam/openrouter-web#47022). Without this change, a run against the alias falls back to high and sends reasoning.effort: high. The router treats that as an effort the user picked, which narrows Jev's candidate pool. Found by Devin Review on that PR. This fix has to land here first because packages/bench-harness there is a vendored subtree.
How to test
parseArgs(["--model", "typesafe/jev-router"]).reasoningEffort returns "auto", so the request is sent without a reasoning field.
Benchmark impact
Default runs against typesafe/jev-router let Jev choose the effort instead of pinning high. openrouter/jev and other models are unaffected.
Reviewer focus
The alias list is a string literal here, since the harness doesn't import openrouter-web's RouterModel.
Checklist
Tests cover changed behavior
Public API or configuration changes are backward compatible, or the break is documented
Benchmark changes document dataset provenance and licensing
No credentials, private results, or restricted dataset contents are included
I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.
Disable automatic comment, CI, and merge conflict monitoring
Original prompt from Ayush
SYSTEM:
<latest_message>
Ayush Patel (U0B8L6RNMA9) [ts=1790354995.883349]: @Devin can you have openrouter/jev also work behind typesafe/jev-router in openrouter-web code
</latest_message>
The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Benchmark runs against
typesafe/jev-routerdefault to adaptive reasoning effort, matchingopenrouter/jev.What changed?
typesafe/jev-routertoADAPTIVE_REASONING_EFFORT_MODELS.--reasoning-effort auto.Why?
openrouter-web is adding
typesafe/jev-routeras an alias for the Jev router (OpenRouterTeam/openrouter-web#47022). Without this change, a run against the alias falls back tohighand sendsreasoning.effort: high. The router treats that as an effort the user picked, which narrows Jev's candidate pool. Found by Devin Review on that PR. This fix has to land here first becausepackages/bench-harnessthere is a vendored subtree.How to test
parseArgs(["--model", "typesafe/jev-router"]).reasoningEffortreturns"auto", so the request is sent without areasoningfield.Benchmark impact
Default runs against
typesafe/jev-routerlet Jev choose the effort instead of pinninghigh.openrouter/jevand other models are unaffected.Reviewer focus
RouterModel.Checklist
Link to Devin session: https://openrouter.devinenterprise.com/sessions/da2afc8a19894fbea2b1bdd33069012d
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/da2afc8a19894fbea2b1bdd33069012d?variant=devin
Requested by: @ayush-or