diff --git a/README.md b/README.md index 76db047..aab88c7 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ E. O'Leary · M. Alshehri · L. Legon --- -> **Abstract.** Frontier language models write Opentrons OT-2 Python that runs in the simulator, but wet-lab correctness depends on tacit knowledge the simulator does not check: which cells must not be pipette-mixed, how much eluate to leave behind with the beads, how much liquid a reservoir well can hold. **Text2WetLab** is a set of 11 [Harbor](https://github.com/laude-institute/harbor) tasks at two levels: *easy* tasks give the exact steps; *paper-only* ("hard") tasks give only a goal, a fixed deck and the source paper. A layered verifier scores each protocol with a lint gate, 10 reward-hacking traps, the Opentrons simulator, deterministic checks against a ground-truth protocol IR, and a pass/fail LLM rubric (majority of three judge calls) in which three core items carry 75% of the reward. We validate the verifier with every reference solution (11/11 pass) and 152 broken or cheating protocols of 18 kinds (none scores above 0.47), fix six grader faults that the model runs exposed, and audit the paper-only tasks so that every quantity they grade is in the paper or the brief, which found and fixed two more. Running the same Claude Code agent with four models through OpenRouter, GPT-6.1 Sol scores 0.881, Claude Sonnet 5.5 0.863, Opus 5.5 0.795 and Fable 5.1 0.705, but Opus and Fable lose 5 trials to Anthropic safety refusals on molecular-cloning prompts; on the 8 tasks every model answered the order is Opus 1.000, Fable 0.969, GPT 0.898, Sonnet 0.842. No model failed the simulator or a trap, and the only physical-safety failure is one reservoir overdraw. Easy tasks are nearly solved; the paper-only tasks separate the models, and the commonest failure is inventing steps and quantities the paper does not support. Five of the 44 scores are provisional until a re-judge with the audited verifier (§6). +> **Abstract.** Frontier language models write Opentrons OT-2 Python that runs in the simulator, but wet-lab correctness depends on tacit knowledge the simulator does not check: which cells must not be pipette-mixed, how much eluate to leave behind with the beads, how much liquid a reservoir well can hold. **Text2WetLab** is a set of 11 [Harbor](https://github.com/laude-institute/harbor) tasks at two levels: *easy* tasks give the exact steps; *paper-only* ("hard") tasks give only a goal, a fixed deck and the source paper. A layered verifier scores each protocol with a lint gate, 10 reward-hacking traps, the Opentrons simulator, deterministic checks against a ground-truth protocol IR, and a pass/fail LLM rubric (majority of three judge calls) in which three core items carry 75% of the reward. We validate the verifier with every reference solution (11/11 pass) and 152 broken or cheating protocols of 18 kinds (none scores above 0.47), fix six grader faults that the model runs exposed, and audit the paper-only tasks so that every quantity they grade is in the paper or the brief, which found and fixed two more. Running the same Claude Code agent with four models through OpenRouter, and scoring only the tasks a model answered (a safety refusal is the provider's policy, not a protocol), Claude Opus 5.5 scores 1.000 on the 8 tasks every model answered, Fable 5.1 0.969, GPT-6.1 Sol 0.898 and Sonnet 5.5 0.842; Opus and Fable refused 5 molecular-cloning prompts between them, which are reported but not scored. No model failed the simulator or a trap, and the only physical-safety failure is one reservoir overdraw. Easy tasks are nearly solved; the paper-only tasks separate the models, and the commonest failure is inventing steps and quantities the paper does not support. Five of the 44 scores are provisional until a re-judge with the audited verifier (§6). --- @@ -252,17 +252,16 @@ The agent gets the instruction, `/data` and a shell, and must leave `/app/protoc
-**Table 6.** Summary. Agent cost is for all 11 trials. † Includes provisional scores (§6). +**Table 6.** Summary. Refused tasks are not scored; the first row is the headline. Agent cost is for every trial run. † Includes provisional scores (§6). | | Sonnet 5.5 | Opus 5.5 | Fable 5.1 | GPT-6.1 Sol | |---|:---:|:---:|:---:|:---:| -| Mean reward, refusals as 0 (n = 11) | 0.863 | 0.795 | 0.705 | **0.881** | -| Mean over tasks the model answered | 0.863 (11) | **0.972** (9) | 0.969 (8) | 0.881 (11) | -| Mean over the 8 tasks all models answered | 0.842 | **1.000** | 0.969 | 0.898 | +| **Mean over the 8 tasks all models answered** | 0.842 | **1.000** | 0.969 | 0.898 | +| Mean over all tasks the model answered | 0.863 (11) | **0.972** (9) | 0.969 (8) | 0.881 (11) | | Easy tasks, answered | 0.900 (7) | **1.000** (6) | 0.958 (6) | **1.000** (7) | | Paper-only tasks, answered | 0.797 (4)† | 0.917 (3) | **1.000** (2) | 0.672 (4)† | | `fidelity_to_paper` passed (hard) | 1/4 | 2/3 | 2/2 | 0/4 | -| Safety refusals | **0** | 2 | 3 | **0** | +| Refused, not scored | 0 | 2 | 3 | 0 | | Simulator failures, traps, lint violations | 0 | 0 | 0 | 0 | | Agent cost (USD) | **1.25** | 3.52 | 6.21 | 4.82 | @@ -272,13 +271,13 @@ Six of the seven easy tasks are saturated: every model that answered them scored ### 5.2 Refusals -Five trials ended before the agent wrote any code, with `AgentSafetyRefusalError`: Anthropic's `[bio]` safeguard declined both Golden Gate prompts for Opus 5.5 and Fable 5.1, and the paper-only E. coli transformation prompt for Fable 5.1. Sonnet 5.5 and GPT-6.1 Sol answered all 11. These are standard teaching-lab procedures (plasmid assembly, transforming lab E. coli), so the refusals are false positives, but they are a property of the deployed model and we report them as they happened; we did not rephrase prompts to get around the filter. +Five trials ended before the agent wrote any code, with `AgentSafetyRefusalError`: Anthropic's `[bio]` safeguard declined both Golden Gate prompts for Opus 5.5 and Fable 5.1, and the paper-only E. coli transformation prompt for Fable 5.1. Sonnet 5.5 and GPT-6.1 Sol answered all 11. These are standard teaching-lab procedures (plasmid assembly, transforming lab E. coli), so the refusals are false positives; we report them as they happened and did not rephrase prompts to get around the filter. **A refused task is not scored.** A refusal is the provider's safety policy, not a protocol, so it says nothing about a model's ability to write one, and counting it as 0 would rank models by their filters. Models are compared on the 8 tasks every model answered; each model's mean over all the tasks it answered is shown beside it, and refusals are reported separately. -

Mean reward under three ways of counting refusals, and cost against reward

+

Mean reward over the tasks all models answered, all tasks answered, easy and paper-only; cost against reward

-

Figure 7 | Refusals change the ranking. a, Mean reward per model with refusals counted as 0, over the tasks each model answered, and over the 8 tasks all four answered. Counting refusals as failures ranks GPT-6.1 Sol and Sonnet first; comparing like with like ranks Opus first. b, Agent cost for the 11 trials against mean reward (refusals as 0). Refused trials cost almost nothing, so Fable's $6.21 is for 8 answered tasks.

+

Figure 7 | Scores without refusals. a, Mean reward over the 8 tasks all four models answered (the headline), over every task each model answered, and over the easy and paper-only tasks it answered. b, Agent cost for the tasks each model answered against its headline score. Refused trials cost almost nothing, so Fable's $6.21 is for 8 tasks.

-Which mean to quote depends on the question. For "which model can I hand this lab's protocols to", refusals are failures and GPT-6.1 Sol leads, with Sonnet close behind at a quarter of the cost. For "which model writes better protocols when it answers", the like-for-like comparison applies and Opus leads, on 8 tasks and one attempt each. +On the 8 tasks all four answered, Opus scores 1.000 and Fable 0.969, ahead of GPT-6.1 Sol (0.898) and Sonnet (0.842). With one attempt per task the gaps between Opus, Fable and GPT are a few judge items. Sonnet is the cheapest by far ($1.25 for 11 tasks), and Sonnet and GPT are the only models that answered every task, which matters to a lab that needs all of its protocols automated even though it is not part of the score. ### 5.3 Per risk: where points were and were not lost @@ -320,7 +319,7 @@ Every failed check and rubric item, with each judge vote and its evidence, is in ## 6. Discussion and limitations - **The paper is the hard part.** Easy tasks are nearly saturated; the paper-only tasks separate models, and the dominant failure there is inventing steps or quantities the paper does not support. GPT-6.1 Sol, perfect on easy tasks, fails `fidelity_to_paper` on all four paper-only ones (two provisional). The paper-only level is the one to grow, and every new task has to pass the audit first. -- **Refusals confound capability.** Two of four models refused standard molecular-cloning protocols. Any leaderboard on this benchmark has to say how it counts refusals (§5.2). +- **Refusals are not scored, but they are not free.** Two of four models refused standard molecular-cloning protocols. We exclude refused tasks from every score (§5.2), which makes the headline a comparison on 8 tasks, not 11; every new model can only shrink that set, so we also report each model's mean over all the tasks it answered, and its refusals. - **Judge noise is real and now measured.** Three calls per protocol disagreed on 7 items across the run (each vote is in the grader record), and a single call had scored identical colony-PCR recipes differently. The majority vote absorbs this, but all signal still comes from one judge, a Claude model that also judges a non-Claude model; a second judge from another provider would show whether that matters. - **Ground truth comes from the easy task.** Three of the four paper-only tasks reuse their easy task's ground truth. That is only fair where the paper fixes the same quantities; the audit now enforces it, and it is why colony-PCR-hard no longer grades its reaction volume deterministically. - **Provisional scores.** Five scores, listed with reasons in [`PROVISIONAL.json`](results/runs/2026-10-07-openrouter/PROVISIONAL.json), were judged before the audit or with fewer than three votes: Sonnet and GPT on RNA-hard and colony-PCR-hard, and Sonnet on ecoli-hard (one vote). They are marked † on the leaderboard. The OpenRouter key ran out of credit before they could be re-judged; `python scripts/regrade_jobs.py --judge --all` does it for about $2. diff --git a/docs/preprint/figures/run_means_cost.png b/docs/preprint/figures/run_means_cost.png index 3fd60a6..57d706d 100644 Binary files a/docs/preprint/figures/run_means_cost.png and b/docs/preprint/figures/run_means_cost.png differ diff --git a/docs/preprint/figures/run_rewards.png b/docs/preprint/figures/run_rewards.png index 2971c6e..f69aa21 100644 Binary files a/docs/preprint/figures/run_rewards.png and b/docs/preprint/figures/run_rewards.png differ diff --git a/docs/preprint/figures/run_where_lost.png b/docs/preprint/figures/run_where_lost.png index d26d9e9..d14e6eb 100644 Binary files a/docs/preprint/figures/run_where_lost.png and b/docs/preprint/figures/run_where_lost.png differ diff --git a/docs/preprint/make_run_figures.py b/docs/preprint/make_run_figures.py index 69bfb78..f093429 100644 --- a/docs/preprint/make_run_figures.py +++ b/docs/preprint/make_run_figures.py @@ -73,16 +73,17 @@ def xgrid(ax): ax.xaxis.grid(True, color=GRID, lw=0.6); ax.set_axisbelow(True) ax.tick_params(length=0); [s.set_visible(False) for s in ax.spines.values()] ax.text(len(MODELS) - .4, (len(EASY) - 1) / 2, "EASY\nstep by step", rotation=90, va="center", ha="left", fontsize=7.5, color=MUTED) ax.text(len(MODELS) - .4, len(EASY) + (len(HARD) - 1) / 2, "HARD\npaper only", rotation=90, va="center", ha="left", fontsize=7.5, color=MUTED) -foot = " · ".join(f"{NAME[m]}: {mean([reward(m, k) if not refused(m, k) else 0 for k in TASKS]):.3f}" for m in MODELS) -ax.text((len(MODELS) - 1) / 2, len(rows) - .1, "Mean, refusals as 0: " + foot, ha="center", va="top", fontsize=8, color=INK2) +foot = " · ".join(f"{NAME[m]}: {mean([reward(m, k) for k in common]):.3f}" for m in MODELS) +ax.text((len(MODELS) - 1) / 2, len(rows) - .1, f"Mean over the {len(common)} tasks all answered: " + foot, ha="center", va="top", fontsize=8, color=INK2) fig.savefig(OUT / "run_rewards.png"); plt.close(fig) # Figure 8: how the mean depends on refusals, and what it cost fig, (a, b) = plt.subplots(1, 2, figsize=(9.2, 3.4), gridspec_kw={"width_ratios": [1.35, 1]}) -rules = [("Refusals count as 0", lambda m: mean([0 if refused(m, k) else reward(m, k) for k in TASKS])), - ("Tasks the model answered", lambda m: mean([reward(m, k) for k in answered[m]])), - (f"The {len(common)} tasks all answered", lambda m: mean([reward(m, k) for k in common]))] +rules = [(f"The {len(common)} tasks\nall answered", lambda m: mean([reward(m, k) for k in common])), + ("All tasks the\nmodel answered", lambda m: mean([reward(m, k) for k in answered[m]])), + ("Easy tasks\nanswered", lambda m: mean([reward(m, k) for k in answered[m] if k in EASY])), + ("Paper-only tasks\nanswered", lambda m: mean([reward(m, k) for k in answered[m] if k in HARD]))] w = .8 / len(MODELS) for j, m in enumerate(MODELS): vals = [f(m) for _, f in rules] @@ -92,18 +93,18 @@ def xgrid(ax): ax.xaxis.grid(True, color=GRID, lw=0.6); ax.set_axisbelow(True) a.text(x, v - .012, f"{v:.2f}", ha="center", va="top", fontsize=6.5, color="white", fontweight="bold") a.set_xticks(range(len(rules)), [r for r, _ in rules], fontsize=8) a.set_ylim(.5, 1.0); a.set_ylabel("Mean reward"); ygrid(a) -a.legend(loc="upper center", bbox_to_anchor=(.5, -.12), ncol=len(MODELS), fontsize=7.5) -a.set_title("Ranking flips with how refusals are counted"); panel(a, "a") +a.legend(loc="upper center", bbox_to_anchor=(.5, -.2), ncol=len(MODELS), fontsize=7.5) +a.set_title("Refused tasks are not scored"); panel(a, "a") for m in MODELS: cost = sum(trials[(m, k)].get("cost_usd") or 0 for k in TASKS) v = rules[0][1](m) b.scatter(cost, v, s=90, color=COL[m], zorder=3) nref = sum(refused(m, k) for k in TASKS) - b.annotate(f"{NAME[m]}\n{nref} refusal{'s' * (nref != 1)}", (cost, v), xytext=(8, -4), textcoords="offset points", fontsize=8, color=INK2) -b.set_xlabel("Agent cost for all 11 tasks (USD)"); b.set_ylabel("Mean reward (refusals as 0)") + b.annotate(f"{NAME[m]}\n{nref} refused, not scored" if nref else NAME[m], (cost, v), xytext=(8, -4), textcoords="offset points", fontsize=8, color=INK2) +b.set_xlabel("Agent cost for the tasks it answered (USD)"); b.set_ylabel(f"Mean over the {len(common)} tasks all answered") costs = [sum(trials[(m, k)].get("cost_usd") or 0 for k in TASKS) for m in MODELS] vals = [rules[0][1](m) for m in MODELS] -b.set_xlim(0, max(costs) * 1.3); b.set_ylim(min(vals) - .05, min(1.0, max(vals) + .05)); ygrid(b); xgrid(b) +b.set_xlim(0, max(costs) * 1.45); b.set_ylim(min(vals) - .05, min(1.02, max(vals) + .05)); ygrid(b); xgrid(b) b.set_title("Cost against reward"); panel(b, "b") fig.tight_layout(); fig.savefig(OUT / "run_means_cost.png"); plt.close(fig) diff --git a/docs/preprint/preprint.html b/docs/preprint/preprint.html index b2b6565..6acb2d1 100644 --- a/docs/preprint/preprint.html +++ b/docs/preprint/preprint.html @@ -117,16 +117,17 @@

Abstract

close reading of the model runs exposed six grader faults, and an audit requiring every quantity a paper-only task grades to be in its paper or brief found two more; all eight are fixed. Running one agent harness -(Claude Code) with four models through one API, GPT-6.1 Sol scores -0.881, Claude Sonnet 5.5 0.863, Opus 5.5 0.795 and Fable 5.1 0.705, but -Opus and Fable lose five trials to safety refusals on standard -molecular-cloning prompts; on the eight tasks every model answered the -order reverses (Opus 1.000, Fable 0.969, GPT 0.898, Sonnet 0.842). No -model failed the simulator or a trap, and only one broke a physical -safety rule. Easy tasks are nearly solved; the paper-only tasks separate -the models, and their commonest failure is inventing steps and -quantities that the paper does not support. Five of the 44 scores are -provisional until a re-judge with the audited verifier.

+(Claude Code) with four models through one API, and scoring only the +tasks a model answered, Claude Opus 5.5 scores 1.000 on the eight tasks +every model answered, Fable 5.1 0.969, GPT-6.1 Sol 0.898 and Sonnet 5.5 +0.842. Opus and Fable refused five standard molecular-cloning prompts +between them; a refusal is the provider’s safety policy, not a protocol, +so these are reported but not scored. No model failed the simulator or a +trap, and only one broke a physical safety rule. Easy tasks are nearly +solved; the paper-only tasks separate the models, and their commonest +failure is inventing steps and quantities that the paper does not +support. Five of the 44 scores are provisional until a re-judge with the +audited verifier.

1 Introduction

Automating a published protocol on a liquid-handling robot is a translation problem with two failure modes. The first is mechanical: @@ -641,14 +642,15 @@

6 Experimental setup

with the benchmark.

7 Results

-Figure 3. Reward on all 11 tasks, one attempt per task and model. Hatched cells are refusals. Above the line, easy tasks; below, paper-only tasks. +Figure 3. Reward on all 11 tasks, one attempt per task and model. Hatched cells are refusals. Above the line, easy tasks; below, paper-only tasks.
-

Table 4. Summary. Agent cost is for all 11 trials. † -Includes provisional scores (Section 8).

+

Table 4. Summary. Refused tasks are not scored; the +first row is the headline. † Includes provisional scores (Section +8).

@@ -668,27 +670,20 @@

7 Results

- - - - - + + + + + - + - - - - - - - @@ -710,11 +705,11 @@

7 Results

- - + + - + @@ -731,22 +726,27 @@

7 Results

and Opus were perfect on every easy task they answered. The models separate on RNA extraction and on the hard tasks, where 8 of 13 answered trials failed fidelity_to_paper.

-

Refusals change the ranking. Anthropic’s biosecurity -safeguard declined both Golden Gate prompts for Opus and Fable and the -paper-only E. coli transformation prompt for Fable. These are -teaching-lab procedures, so the refusals are false positives, but they -are a property of the deployed model; we report them as they happened -and did not rephrase prompts. Counted as failures, they put GPT-6.1 Sol -and Sonnet first; excluded, Opus leads (Figure 4). ABC-Bench reports -refusals on its dual-use screening-evasion task, where the tested -Anthropic and OpenAI frontier models refused every sample (A. B. Liu et al. +

Refused tasks are not scored. Anthropic’s +biosecurity safeguard declined both Golden Gate prompts for Opus and +Fable and the paper-only E. coli transformation prompt for Fable. These +are teaching-lab procedures, so the refusals are false positives; we +report them as they happened and did not rephrase prompts. A refusal is +the provider’s policy, not a protocol, and says nothing about a model’s +ability to write one, so we compare models on the eight tasks all four +answered and report refusals separately (Figure 4). Opus leads there, +with Fable, GPT and Sonnet a few judge items behind; Sonnet is the +cheapest, and Sonnet and GPT are the only models that answered every +task. ABC-Bench reports refusals on its dual-use screening-evasion task, +where the tested Anthropic and OpenAI frontier models refused every +sample (A. B. Liu et al. 2026); ours fall on benign tasks.

-Figure 4. (a) Mean reward with refusals counted as 0, over the tasks each model answered, and over the eight tasks all four answered. (b) Agent cost for 11 trials against mean reward. +Figure 4. (a) Mean reward over the eight tasks all four models answered (the headline), over every task each model answered, and over the easy and paper-only tasks it answered. (b) Agent cost for the tasks each model answered against its headline score. +reward over the eight tasks all four models answered (the headline), +over every task each model answered, and over the easy and paper-only +tasks it answered. (b) Agent cost for the tasks each model answered +against its headline score.

The mechanical layers are solved; the signal is in the judge. No model crashed the simulator, tripped a trap or broke @@ -755,7 +755,7 @@

7 Results

plate for every model, which built the paper’s 10 µL reaction where the ground truth fixes 20 µL, and GPT’s Golden-Gate-hard plate.

-Figure 5. (a) Deterministic checks passed per risk and model. (b) Every rubric item the judge failed: 14 failures in 39 judged trials; tips_and_contamination passed in all 39. +Figure 5. (a) Deterministic checks passed per risk and model. (b) Every rubric item the judge failed: 14 failures in 39 judged trials; tips_and_contamination passed in all 39.
Mean reward, refusals as 00.8630.7950.7050.881Mean over the 8 tasks all answered0.8421.0000.9690.898
Mean over tasks answeredMean over all tasks answered 0.863 (11) 0.972 (9) 0.969 (8) 0.881 (11)
Mean over the 8 tasks all answered0.8421.0000.9690.898
Easy tasks, answered 0.900 1.000 0/4
Safety refusals0Refused, not scored0 2 300
Agent cost (USD)