Skip to content

refactor: rename the native benchmark set to Kepler - #120

Merged
abhinav-pola merged 2 commits into
mainfrom
devin/1790381704-mendel-rename
Sep 26, 2026
Merged

abhinav-pola merged 2 commits into
mainfrom
devin/1790381704-mendel-rename

Conversation

@abhinav-pola

@abhinav-pola abhinav-pola commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

The TS benchmark harness is now called Kepler. This PR renames the harness-specific "native" identifiers to match.

What changed?

  • NativeBenchmarkRunConfigSchema / NativeBenchmarkRunConfig → KeplerBenchmarkRunConfigSchema / KeplerBenchmarkRunConfig
  • isNativeBenchmarkConfig → isKeplerBenchmarkConfig, and the private NATIVE_BENCHMARK_ID_SET / NativeModelBenchmarkConfig get the same rename
  • run-by-id renames nativeBenchmark → keplerBenchmark, and the injected-id error messages now say "Kepler benchmark"
  • package.json description updated

Not changed, because these uses of "native" mean something else:

  • WebSearchEngine.Native and the draco/search "native" engine values, which are OpenRouter's native web search
  • @typescript/native
  • bench-worker-native in the logger tests, which is the deployed Cloud Run resource name and will be renamed separately if at all

Why?

"Native" was too generic a name for the harness, so it now has a proper noun (Kepler), alongside Harbor.

How to test

rg -n 'NativeBenchmarkRunConfig|isNativeBenchmarkConfig' src returns nothing. There is no behavior change.

Reviewer focus

  • This is a breaking rename of public exports with no aliases. Nothing in the openrouter-web subtree consumer imports these names outside packages/bench-harness, so the next subtree pull needs no follow-up edits.

Checklist

  • Tests cover changed behavior
  • Public API or configuration changes are backward compatible, or the break is documented
  • Benchmark changes document dataset provenance and licensing (n/a)
  • No credentials, private results, or restricted dataset contents are included
  • Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/26aed72e72b94648bcafa700baf71832
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/26aed72e72b94648bcafa700baf71832?variant=devin
Requested by: @abhinav-pola


Devin Review

@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

Original prompt from Abhinav

SYSTEM:
<latest_message>
Abhinav Pola (U090K0G7JF3) [ts=1790381041.747209]: @Devin instead of "native" we need to call our TS benchmark harness something else. come up with a cool name
</latest_message>

=== BEGIN THREAD HISTORY (in #agents-benchmarks) ===
Abhinav Pola (U090K0G7JF3) [ts=1790381041.747209]: @Devin instead of "native" we need to call our TS benchmark harness something else. come up with a cool name
=== END THREAD HISTORY ===
Channel ID: C0BAKP8P5C3
Thread URL: https://openrouter.slack.com/archives/C0BAKP8P5C3/p1790381041747209?thread_ts=1790381041.747209&amp;cid=C0BAKP8P5C3

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@devin-ai-integration devin-ai-integration Bot changed the title refactor: rename the native benchmark set to Mendel refactor: rename the native benchmark set to Kepler Sep 26, 2026
@abhinav-pola
abhinav-pola merged commit 61205fa into main Sep 26, 2026
5 checks passed
@abhinav-pola
abhinav-pola deleted the devin/1790381704-mendel-rename branch September 26, 2026 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants