Skip to content

refactor(bench): name the sandbox plugin guard and the wire-shape plugin merge - #119

Draft
jamespsterling wants to merge 1 commit into
mainfrom
devin/1790369465-switchyard-harness-review-followup
Draft

jamespsterling wants to merge 1 commit into
mainfrom
devin/1790369465-switchyard-harness-review-followup

Conversation

@jamespsterling

@jamespsterling jamespsterling commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

Two naming follow-ups from Perry's review of openrouter-web#46340 on the harness code vendored there; no behavior change.

What changed?

  • sandboxAgentCandidateModelsError becomes sandboxAgentPluginError (src/benchmarks/agent-cli/candidate-models.ts and its three callers: terminal-bench, deep-swe, swe-atlas). It rejects both candidate models and switchyardAlgorithm for sandbox agents, so the old name covered half the job.
  • The search solver's SDK serialize-and-parse round trip moves into pluginsInWireShape(body). extraBody is spread over the serialized wire body, so the plugins array it carries replaces the SDK's; the existing plugins therefore have to be in wire shape (max_results, not maxResults) before the Switchyard plugin is appended. The repo's openrouter/no-comments lint rule forbids explaining this in a comment, so the function name carries it.

Why?

Review threads on openrouter-web#46340 (rename, round trip) point at files in the vendored subtree, which can only change upstream here. Searched existing PRs; none matched (#97 and #109 introduced the code).

How to test

bun test src/benchmarks/agent-cli src/benchmarks/search

Expected: all tests pass; the sandboxAgentPluginError suite runs three cases (no config, candidate list, algorithm).

How to test in prod
  • What to verify: nothing user-visible changes. After the next subtree pull into openrouter-web, a search benchmark run with switchyardAlgorithm set records the same requestBody.plugins (web plugin in wire shape plus switchyard-router) as before this PR.
  • Runnable verification: in openrouter-web, run bun test packages/bench-harness/src/benchmarks/search/core after the pull; expected 0 fail.
  • Rollback signal: a search run's stored requestBody.plugins missing the web plugin or carrying camelCase keys such as maxResults.

Reviewer focus

  • The extracted pluginsInWireShape returns [] when the serialized body has no plugins array, matching the previous inline fallback.

Checklist

  • Tests cover changed behavior
  • Public API or configuration changes are backward compatible, or the break is documented (internal exports only; no config change)
  • Benchmark changes document dataset provenance and licensing (n/a)
  • No credentials, private results, or restricted dataset contents are included
  • Documentation is updated where needed (n/a)

Link to Devin session: https://openrouter.devinenterprise.com/sessions/37ef700865744be284bb0127c4f6a8d0
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/37ef700865744be284bb0127c4f6a8d0?variant=devin
Requested by: @jamespsterling

…gin merge

Renames sandboxAgentCandidateModelsError to sandboxAgentPluginError, since it
rejects both candidate models and switchyardAlgorithm for sandbox agents, and
extracts pluginsInWireShape in the search solver so the SDK serializer round
trip reads as the wire-shape conversion it is.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

Original prompt from James

Analyze this session: @devin-session:devin-9b303b51c68b4f74a224b28cefa855ae

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant