refactor(bench): name the sandbox plugin guard and the wire-shape plugin merge - #119
Draft
jamespsterling wants to merge 1 commit into
Draft
jamespsterling wants to merge 1 commit into
jamespsterling wants to merge 1 commit into
Conversation
…gin merge Renames sandboxAgentCandidateModelsError to sandboxAgentPluginError, since it rejects both candidate models and switchyardAlgorithm for sandbox agents, and extracts pluginsInWireShape in the search solver so the SDK serializer round trip reads as the wire-shape conversion it is. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
|
I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.
Original prompt from James
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Two naming follow-ups from Perry's review of openrouter-web#46340 on the harness code vendored there; no behavior change.
What changed?
sandboxAgentCandidateModelsErrorbecomessandboxAgentPluginError(src/benchmarks/agent-cli/candidate-models.tsand its three callers: terminal-bench, deep-swe, swe-atlas). It rejects both candidatemodelsandswitchyardAlgorithmfor sandbox agents, so the old name covered half the job.pluginsInWireShape(body).extraBodyis spread over the serialized wire body, so thepluginsarray it carries replaces the SDK's; the existing plugins therefore have to be in wire shape (max_results, notmaxResults) before the Switchyard plugin is appended. The repo'sopenrouter/no-commentslint rule forbids explaining this in a comment, so the function name carries it.Why?
Review threads on openrouter-web#46340 (rename, round trip) point at files in the vendored subtree, which can only change upstream here. Searched existing PRs; none matched (#97 and #109 introduced the code).
How to test
bun test src/benchmarks/agent-cli src/benchmarks/searchExpected: all tests pass; the
sandboxAgentPluginErrorsuite runs three cases (no config, candidate list, algorithm).How to test in prod
searchbenchmark run withswitchyardAlgorithmset records the samerequestBody.plugins(web plugin in wire shape plusswitchyard-router) as before this PR.bun test packages/bench-harness/src/benchmarks/search/coreafter the pull; expected0 fail.searchrun's storedrequestBody.pluginsmissing the web plugin or carrying camelCase keys such asmaxResults.Reviewer focus
pluginsInWireShapereturns[]when the serialized body has nopluginsarray, matching the previous inline fallback.Checklist
Link to Devin session: https://openrouter.devinenterprise.com/sessions/37ef700865744be284bb0127c4f6a8d0
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/37ef700865744be284bb0127c4f6a8d0?variant=devin
Requested by: @jamespsterling