Skip to content

fix: default typesafe/jev-router benchmarks to adaptive effort - #115

Merged
ayush-or merged 1 commit into
mainfrom
devin/1790355884-typesafe-jev-router-adaptive
Sep 25, 2026
Merged

ayush-or merged 1 commit into
mainfrom
devin/1790355884-typesafe-jev-router-adaptive

Conversation

@ayush-or

Copy link
Copy Markdown
Contributor

TL;DR

Benchmark runs against typesafe/jev-router default to adaptive reasoning effort, matching openrouter/jev.

What changed?

  • Added typesafe/jev-router to ADAPTIVE_REASONING_EFFORT_MODELS.
  • Added tests for the alias default and an explicit --reasoning-effort auto.

Why?

openrouter-web is adding typesafe/jev-router as an alias for the Jev router (OpenRouterTeam/openrouter-web#47022). Without this change, a run against the alias falls back to high and sends reasoning.effort: high. The router treats that as an effort the user picked, which narrows Jev's candidate pool. Found by Devin Review on that PR. This fix has to land here first because packages/bench-harness there is a vendored subtree.

How to test

parseArgs(["--model", "typesafe/jev-router"]).reasoningEffort returns "auto", so the request is sent without a reasoning field.

Benchmark impact

Default runs against typesafe/jev-router let Jev choose the effort instead of pinning high. openrouter/jev and other models are unaffected.

Reviewer focus

  • The alias list is a string literal here, since the harness doesn't import openrouter-web's RouterModel.

Checklist

  • Tests cover changed behavior
  • Public API or configuration changes are backward compatible, or the break is documented
  • Benchmark changes document dataset provenance and licensing
  • No credentials, private results, or restricted dataset contents are included
  • Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/da2afc8a19894fbea2b1bdd33069012d
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/da2afc8a19894fbea2b1bdd33069012d?variant=devin
Requested by: @ayush-or

@ayush-or
ayush-or requested a review from a team as a code owner September 25, 2026 17:05
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

Original prompt from Ayush

SYSTEM:
<latest_message>
Ayush Patel (U0B8L6RNMA9) [ts=1790354995.883349]: @Devin can you have openrouter/jev also work behind typesafe/jev-router in openrouter-web code
</latest_message>

=== BEGIN THREAD HISTORY (in #agents-ecosystem) ===
Ayush Patel (U0B8L6RNMA9) [ts=1790354995.883349]: @Devin can you have openrouter/jev also work behind typesafe/jev-router in openrouter-web code
=== END THREAD HISTORY ===
Channel ID: C0BU53A7VEH
Thread URL: https://openrouter.slack.com/archives/C0BU53A7VEH/p1790354995883349?thread_ts=1790354995.883349&amp;cid=C0BU53A7VEH

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

@ayush-or
ayush-or merged commit e73bbaf into main Sep 25, 2026
4 checks passed
@ayush-or
ayush-or deleted the devin/1790355884-typesafe-jev-router-adaptive branch September 25, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant