diff --git a/README.md b/README.md index aab88c7..240b152 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ E. O'Leary · M. Alshehri · L. Legon --- -> **Abstract.** Frontier language models write Opentrons OT-2 Python that runs in the simulator, but wet-lab correctness depends on tacit knowledge the simulator does not check: which cells must not be pipette-mixed, how much eluate to leave behind with the beads, how much liquid a reservoir well can hold. **Text2WetLab** is a set of 11 [Harbor](https://github.com/laude-institute/harbor) tasks at two levels: *easy* tasks give the exact steps; *paper-only* ("hard") tasks give only a goal, a fixed deck and the source paper. A layered verifier scores each protocol with a lint gate, 10 reward-hacking traps, the Opentrons simulator, deterministic checks against a ground-truth protocol IR, and a pass/fail LLM rubric (majority of three judge calls) in which three core items carry 75% of the reward. We validate the verifier with every reference solution (11/11 pass) and 152 broken or cheating protocols of 18 kinds (none scores above 0.47), fix six grader faults that the model runs exposed, and audit the paper-only tasks so that every quantity they grade is in the paper or the brief, which found and fixed two more. Running the same Claude Code agent with four models through OpenRouter, and scoring only the tasks a model answered (a safety refusal is the provider's policy, not a protocol), Claude Opus 5.5 scores 1.000 on the 8 tasks every model answered, Fable 5.1 0.969, GPT-6.1 Sol 0.898 and Sonnet 5.5 0.842; Opus and Fable refused 5 molecular-cloning prompts between them, which are reported but not scored. No model failed the simulator or a trap, and the only physical-safety failure is one reservoir overdraw. Easy tasks are nearly solved; the paper-only tasks separate the models, and the commonest failure is inventing steps and quantities the paper does not support. Five of the 44 scores are provisional until a re-judge with the audited verifier (§6). +> **Abstract.** Frontier language models write Opentrons OT-2 Python that runs in the simulator, but wet-lab correctness depends on tacit knowledge the simulator does not check: which cells must not be pipette-mixed, how much eluate to leave behind with the beads, how much liquid a reservoir well can hold. **Text2WetLab** is a set of 11 [Harbor](https://github.com/laude-institute/harbor) tasks at two levels: *easy* tasks give the exact steps; *paper-only* ("hard") tasks give only a goal, a fixed deck and the source paper. A layered verifier scores each protocol with a lint gate, 10 reward-hacking traps, the Opentrons simulator, deterministic checks against a ground-truth protocol IR, and a pass/fail LLM rubric (majority of three judge calls) in which three core items carry 75% of the reward. We validate the verifier with every reference solution (11/11 pass) and 151 broken or cheating protocols of 18 kinds (none scores above 0.47), fix eight grader faults that the model runs exposed, and audit the paper-only tasks so that every quantity they grade is in the paper or the brief, which found and fixed two more. Running the same Claude Code agent with six models through OpenRouter, and scoring only the tasks a model answered (a safety refusal is the provider's policy, not a protocol), Claude Opus 5.5 scores 1.000 on the 8 tasks every model answered, Fable 5.1 0.969, GPT-6.1 Sol 0.938, Sonnet 5.5 0.881, Qwen3.8-2.4T-A95B 0.832 and DeepSeek V4 Pro 0.755; Opus and Fable refused 5 molecular-cloning prompts between them, which are reported but not scored. No model failed the simulator or a trap. The physical-safety failures are three reservoir overdraws and, in the two open-weight models only, four cross-contaminations from one tip mixing in a reaction and returning to a stock. Easy tasks are nearly solved; the paper-only tasks separate the models, and the commonest failure is inventing steps and quantities the paper does not support. A repeat of one model on Modal shows single-attempt scores moving by up to 0.70 on a task and 0.08 in the mean. --- @@ -185,13 +185,13 @@ A benchmark is only as good as its ground truth, so we tested the verifier from

Matrix of reference controls and attacks per task, with the reward each received

-

Figure 5 | Grader validation. Deterministic reward (judge off) for the reference solution at three API levels (left of the line, must be 1) and 152 attacks on 9 tasks (right, must be below 1). All 27 controls score 1 and no attack does: the best-scoring attack, dropping the last transfer in Golden Gate, gets 0.47. n/a: the edit does not apply to the task, or leaves the simulated run and labware unchanged (one tip per one-well transfer() call on ecoli-hard is already a fresh tip each time). Wrong-science edits get partial credit at most (deterministic reward is at most 0.5 when any check fails); every attack on the grader scores 0. Data: results/adversarial.json.

+

Figure 5 | Grader validation. Deterministic reward (judge off) for the reference solution at three API levels (left of the line, must be 1) and 151 attacks on 9 tasks (right, must be below 1). All 27 controls score 1 and no attack does: the best-scoring attack, dropping the last transfer in Golden Gate, gets 0.47. Outlined: halving every volume on colony-PCR-hard gives 9 + 0.5 + 0.5 = 10 µL, the paper's own OT-2 reaction, so it is a valid protocol and must score 1. n/a: the edit does not apply to the task, or leaves the simulated run and labware unchanged (one tip per one-well transfer() call on ecoli-hard is already a fresh tip each time). Wrong-science edits get partial credit at most (deterministic reward is at most 0.5 when any check fails); every attack on the grader scores 0. Data: results/adversarial.json.

-**What the model runs taught the verifier.** Validation also ran in the other direction: we read every failed check and every failed rubric item from the model runs in §5 and confirmed each against the protocol. That found six faults. A second check asked whether each paper-only task grades only what its paper or brief says (`tests/test_hard_tasks.py`), and found two more. All eight are fixed, each with a regression test (`tests/test_benchmark_regressions.py`, `tests/test_spec_check.py`, `tests/test_hard_tasks.py`): +**What the model runs taught the verifier.** Validation also ran in the other direction: we read every failed check and every failed rubric item from the model runs in §5 and confirmed each against the protocol. That found eight faults, two of them in the open-weight models' runs. A second check asked whether each paper-only task grades only what its paper or brief says (`tests/test_hard_tasks.py`), and found two more. All ten are fixed, each with a regression test (`tests/test_benchmark_regressions.py`, `tests/test_spec_check.py`, `tests/test_hard_tasks.py`):
-**Table 4.** Grader faults found by reading the model runs (rows 1-6) and by the paper-only audit (rows 7-8). +**Table 4.** Grader faults found by reading the model runs (rows 1-6, and 9-10 from the open-weight runs) and by the paper-only audit (rows 7-8). | Fault | Found in | Fix | |---|---|---| @@ -202,7 +202,9 @@ A benchmark is only as good as its ground truth, so we tested the verifier from | One judge call is noisy: identical colony-PCR recipes passed for one model and failed for two | Sonnet, Opus, Fable, colony-hard | Three judge calls per protocol, majority per item; the rubric says the primer volume is the agent's to choose, as the deck gives no primer concentration | | End-state details were cut to 160 characters, hiding the measured volumes | Every colony-hard report | Full expected and measured values recorded | | RNA-hard's rubric required about 80 µL recovered; the paper says only "After 90 sec collect the supernatant", and the 80 µL is in the authors' code | Sonnet and GPT marked down for recovering the whole 100 µL | Hard rubric accepts any full recovery (70–100 µL); the 80 µL check runs on the easy task only | -| Colony-PCR-hard's end-state check used the easy task's 20 µL reaction; the paper's OT-2 reaction is 10 µL and the deck gives no primer concentration | Every model "failed" it, and the judge saw that as evidence | `not_from_paper = ["pcr_plate"]`: the reaction volume is judged by the rubric only; deck, tip and contamination checks stay | +| Colony-PCR-hard's end-state check used the easy task's 20 µL reaction; the paper's OT-2 reaction is 10 µL and the deck gives no primer concentration | Every model "failed" it, and the judge saw that as evidence | `not_from_paper = { pcr_plate = [10, 25] }`: no exact volumes; instead every well must get master mix, its primers and its colony, all wells alike, 10–25 µL (the paper's OT-2 reaction up to a standard Q5 reaction) | +| The lint gate refused any assignment to an attribute, including Opentrons' documented `pipette.tip_racks = [...]` | DeepSeek, `a1-a12-100ul`: a correct protocol scored 0 | Documented pipette settings (`tip_racks`, `starting_tip`, `default_speed`, `flow_rate.*`, `well_bottom_clearance.*`) allowed; patching any class or module is still refused | +| The contamination check flagged a tip that mixed in a source well before drawing from it (resuspending a colony) | DeepSeek, colony-hard: capped at 0.30, really 1.00 | Only mixing in a destination makes a tip carry that well's contents |
@@ -215,11 +217,11 @@ to follow from quoted facts by arithmetic the test spells out. After the two fix | `ecoli-heat-shock-transformation-hard` | Its own IR, from the APEX paper | Paper text (4 °C 30 min, 42 °C 30 s, 37 °C 1 h) and a figure caption (1 µL DNA, 10 µL cells, 50 µL SOC) | | `golden-gate-assembly-hard` | The easy task's IR | Paper ("25 µL volumes … 0.1 µM primers and 0.5 ng … template"), the brief's design table and stock concentrations, and the manufacturer's standard recipe: 2.5 µL primers, 19 µL master mix, 106/40/4/2 µL for 8 reactions, 7 µL water to 20 µL | | `opentrons-rna-extraction-hard` | The easy task's run-log checks, minus the 80 µL | Paper (40/250/250 µL, 2 × 500 µL 70% ethanol, 4 min, 100 µL, 30 s, 90 s) and the brief (4 °C plate) | -| `colony-pcr-screening-hard` | The easy task's IR, without the PCR-plate volumes | Deck, tip and contamination checks only; the reaction is judged against the paper's 10 µL or the master mix maker's 20-25 µL | +| `colony-pcr-screening-hard` | The easy task's IR, without the exact PCR-plate volumes | Paper ("9 μL of this master mix" + 1 µL colony = 10 µL) for the lower bound of a 10–25 µL reaction with all three inputs in every well; deck, tip and contamination checks as usual | Whether the level is *harder* is a weaker claim. Its briefs list no steps and few volumes (RNA-hard names 2, the easy -brief 10), so the agent must extract the procedure; but each model has only 2-4 paper-only trials, and five of them -are provisional. We call it paper-only rather than hard for that reason. +brief 10), so the agent must extract the procedure, and every model but Fable scores lower on it than on the easy +tasks; but each model has only 2-4 paper-only trials, one attempt each. We call it paper-only rather than hard for that reason. ## 4. Experimental setup @@ -230,13 +232,14 @@ are provisional. We call it paper-only rather than hard for that reason. | | | |---|---| | Tasks | all 11 (7 easy, 4 hard) | -| Trials | 44: 4 models × 11 tasks, one attempt each (pass@1) | +| Trials | 66: 6 models × 11 tasks, one attempt each (pass@1); plus an 11-task repeat of Qwen on Modal | | Agent | Claude Code 2.1.288 (`-a claude-code`) for every model, via OpenRouter's Anthropic-compatible API; Harbor pins every model alias and sub-agent to the model under test, and every transcript was checked to contain only that model | -| Models | Claude Sonnet 5.5, Opus 5.5 and Fable 5.1 (Anthropic); GPT-6.1 Sol (OpenAI) | +| Models | Claude Sonnet 5.5, Opus 5.5 and Fable 5.1 (Anthropic); GPT-6.1 Sol (OpenAI); Qwen3.8-2.4T-A95B (Alibaba, open weights); DeepSeek V4 Pro (DeepSeek, open weights) | | Judge | Claude Sonnet 5.5 via OpenRouter, three calls per protocol, majority per item | | Rubric | 3 core items at 75%, task items at 25% (§3.2) | -| Sandbox | Harbor 0.23.0, Docker (Colima), 2 CPUs, 2.5 GB | -| Verifier | The Claude trials were regraded from their saved protocols with the final verifier (`scripts/regrade_jobs.py --judge --all`, no agent rerun); the GPT trials were graded by it directly | +| Sandbox | Harbor 0.23.0, Docker (Colima), 2 CPUs, 2.5 GB, two trials at a time; DeepSeek on Modal (`-e modal`), all 11 at once | +| Verifier | Every trial is graded by the final verifier: earlier trials were regraded from their saved protocols (`scripts/regrade_jobs.py --judge --all`, no agent rerun) | +| Cost | Agent tokens at OpenRouter prices (`prices.json`); Claude Code's own figure prices every model as Claude, so it is kept only as `claude_code_cost_usd` | @@ -252,78 +255,106 @@ The agent gets the instruction, `/data` and a shell, and must leave `/app/protoc
-**Table 6.** Summary. Refused tasks are not scored; the first row is the headline. Agent cost is for every trial run. † Includes provisional scores (§6). +**Table 6.** Summary. Refused tasks are not scored; the first row is the headline. Agent cost is for every trial run. -| | Sonnet 5.5 | Opus 5.5 | Fable 5.1 | GPT-6.1 Sol | -|---|:---:|:---:|:---:|:---:| -| **Mean over the 8 tasks all models answered** | 0.842 | **1.000** | 0.969 | 0.898 | -| Mean over all tasks the model answered | 0.863 (11) | **0.972** (9) | 0.969 (8) | 0.881 (11) | -| Easy tasks, answered | 0.900 (7) | **1.000** (6) | 0.958 (6) | **1.000** (7) | -| Paper-only tasks, answered | 0.797 (4)† | 0.917 (3) | **1.000** (2) | 0.672 (4)† | -| `fidelity_to_paper` passed (hard) | 1/4 | 2/3 | 2/2 | 0/4 | -| Refused, not scored | 0 | 2 | 3 | 0 | -| Simulator failures, traps, lint violations | 0 | 0 | 0 | 0 | -| Agent cost (USD) | **1.25** | 3.52 | 6.21 | 4.82 | +| | Sonnet 5.5 | Opus 5.5 | Fable 5.1 | GPT-6.1 Sol | Qwen3.8-2.4T | DeepSeek V4 Pro | +|---|:---:|:---:|:---:|:---:|:---:|:---:| +| **Mean over the 8 tasks all models answered** | 0.881 | **1.000** | 0.969 | 0.938 | 0.832 | 0.755 | +| Mean over all tasks the model answered | 0.891 (11) | **0.972** (9) | 0.969 (8) | 0.909 (11) | 0.728 (11) | 0.735 (11) | +| Easy tasks, answered | 0.900 (7) | **1.000** (6) | 0.958 (6) | **1.000** (7) | 0.800 (7) | 0.800 (7) | +| Paper-only tasks, answered | 0.875 (4) | 0.917 (3) | **1.000** (2) | 0.750 (4) | 0.601 (4) | 0.622 (4) | +| `fidelity_to_paper` passed (hard) | 2/4 | 2/3 | 2/2 | 1/4 | 0/4 | 2/4 | +| Refused, not scored | 0 | 2 | 3 | 0 | 0 | 0 | +| Critical failures (reward capped at 0.30) | 1 | 0 | 0 | 0 | 3 | 3 | +| Simulator failures, traps, lint violations | 0 | 0 | 0 | 0 | 0 | 0 | +| Agent cost (USD) | **1.25** | 3.52 | 6.21 | 1.49 | 2.83 | 2.53 |
-Six of the seven easy tasks are saturated: every model that answered them scored 1.0. The models separate on RNA extraction and on the hard tasks, where 8 of 13 answered trials failed `fidelity_to_paper` (§5.4). GPT-6.1 Sol is perfect on the easy tasks and weakest on the hard ones (0.672), where it adds steps of its own; Fable is perfect on the two hard tasks it answered but refused the other two. With one attempt per cell, single-task differences of 0.25 (one core item) are within run-to-run noise. +Four of the seven easy tasks are saturated: every model scored 1.0 on them. The models separate on RNA extraction and on the hard tasks, where 12 of 21 answered trials failed `fidelity_to_paper` (§5.4). GPT-6.1 Sol is perfect on the easy tasks and weaker on the hard ones (0.750), where it adds steps of its own; Fable is perfect on the two hard tasks it answered but refused the other two. The two open-weight models trail on both levels, and they account for six of the seven critical failures: they mix with a tip and send it back to a stock, and they overdraw the RNA ethanol reservoir. With one attempt per cell, single-task differences of 0.25 (one core item) are within run-to-run noise (§5.5). ### 5.2 Refusals -Five trials ended before the agent wrote any code, with `AgentSafetyRefusalError`: Anthropic's `[bio]` safeguard declined both Golden Gate prompts for Opus 5.5 and Fable 5.1, and the paper-only E. coli transformation prompt for Fable 5.1. Sonnet 5.5 and GPT-6.1 Sol answered all 11. These are standard teaching-lab procedures (plasmid assembly, transforming lab E. coli), so the refusals are false positives; we report them as they happened and did not rephrase prompts to get around the filter. **A refused task is not scored.** A refusal is the provider's safety policy, not a protocol, so it says nothing about a model's ability to write one, and counting it as 0 would rank models by their filters. Models are compared on the 8 tasks every model answered; each model's mean over all the tasks it answered is shown beside it, and refusals are reported separately. +Five trials ended before the agent wrote any code, with `AgentSafetyRefusalError`: Anthropic's `[bio]` safeguard declined both Golden Gate prompts for Opus 5.5 and Fable 5.1, and the paper-only E. coli transformation prompt for Fable 5.1. Sonnet 5.5, GPT-6.1 Sol, Qwen and DeepSeek answered all 11. These are standard teaching-lab procedures (plasmid assembly, transforming lab E. coli), so the refusals are false positives; we report them as they happened and did not rephrase prompts to get around the filter. **A refused task is not scored.** A refusal is the provider's safety policy, not a protocol, so it says nothing about a model's ability to write one, and counting it as 0 would rank models by their filters. Models are compared on the 8 tasks every model answered; each model's mean over all the tasks it answered is shown beside it, and refusals are reported separately.

Mean reward over the tasks all models answered, all tasks answered, easy and paper-only; cost against reward

-

Figure 7 | Scores without refusals. a, Mean reward over the 8 tasks all four models answered (the headline), over every task each model answered, and over the easy and paper-only tasks it answered. b, Agent cost for the tasks each model answered against its headline score. Refused trials cost almost nothing, so Fable's $6.21 is for 8 tasks.

+

Figure 7 | Scores without refusals. a, Mean reward over the 8 tasks all six models answered (the headline), over every task each model answered, and over the easy and paper-only tasks it answered. b, Agent cost for the tasks each model answered against its headline score. Refused trials cost almost nothing, so Fable's $6.21 is for 8 tasks.

-On the 8 tasks all four answered, Opus scores 1.000 and Fable 0.969, ahead of GPT-6.1 Sol (0.898) and Sonnet (0.842). With one attempt per task the gaps between Opus, Fable and GPT are a few judge items. Sonnet is the cheapest by far ($1.25 for 11 tasks), and Sonnet and GPT are the only models that answered every task, which matters to a lab that needs all of its protocols automated even though it is not part of the score. +On the 8 tasks all six answered, Opus scores 1.000 and Fable 0.969, ahead of GPT-6.1 Sol (0.938), Sonnet (0.881), Qwen (0.832) and DeepSeek (0.755). With one attempt per task the gaps between Opus, Fable and GPT are a few judge items. Sonnet and GPT are the cheapest ($1.25 and $1.49 for 11 tasks). Four models answered every task, which matters to a lab that needs all of its protocols automated even though it is not part of the score. ### 5.3 Per risk: where points were and were not lost

Deterministic checks passed per risk, and failed rubric items per item

-

Figure 8 | Deterministic layers against the judge. a, Share of deterministic checks passed, per risk and model, over all answered trials. End-state misses are the colony-PCR-hard plate (all models) and GPT's Golden-Gate-hard plate (§5.4); the RNA miss is Sonnet's reservoir overdraw. b, Every rubric item the judge failed, by item and model: 14 failures in 39 judged trials. tips_and_contamination passed in all 39.

+

Figure 8 | Deterministic layers against the judge. a, Deterministic checks passed / run, per risk and model, over all answered trials (red: at least one failed). End-state misses are GPT's and Qwen's Golden-Gate-hard plates; the contamination misses are Qwen's and DeepSeek's shared tips; the RNA misses are reservoir overdraws (§5.4). b, Every rubric item the judge failed, by item and model: 28 failures in 61 judged trials.

**Table 7.** Deterministic checks passed / run, per risk. Each check compares the simulated run with the task's ground truth. -| Risk | Sonnet 5.5 | Opus 5.5 | Fable 5.1 | GPT-6.1 Sol | -|---|:---:|:---:|:---:|:---:| -| Protocol runs in the simulator | 9/9 | 7/7 | 6/6 | 9/9 | -| End-state volumes match the IR | 43/44 | 13/14 | 11/12 | 42/44 | -| Right labware, label and slot | all pass | all pass | all pass | all pass | -| Never pipettes without a tip | 9/9 | 7/7 | 6/6 | 9/9 | -| Never overdispenses | 9/9 | 7/7 | 6/6 | 9/9 | -| Never aspirates from an empty well | 9/9 | 7/7 | 6/6 | 9/9 | -| Drops its tip at the end | 9/9 | 7/7 | 6/6 | 9/9 | -| No cross-contamination | 9/9 | 7/7 | 6/6 | 9/9 | -| No reservoir column overdrawn (RNA) | 1/2 | 2/2 | 2/2 | 2/2 | -| Other RNA run-log checks (16 per trial) | 32/32 | 32/32 | 32/32 | 32/32 | -| Traps tripped / lint violations | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | +| Risk | Sonnet 5.5 | Opus 5.5 | Fable 5.1 | GPT-6.1 Sol | Qwen3.8-2.4T | DeepSeek V4 Pro | +|---|:---:|:---:|:---:|:---:|:---:|:---:| +| Protocol runs in the simulator | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| End-state volumes match the IR (or the paper's range) | 44/44 | 14/14 | 12/12 | 43/44 | 42/44 | 44/44 | +| Right labware, label and slot | all pass | all pass | all pass | all pass | all pass | all pass | +| Never pipettes without a tip | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| Never overdispenses | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| Never aspirates from an empty well | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| Drops its tip at the end | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| No cross-contamination | 9/9 | 7/7 | 6/6 | 9/9 | 7/9 | 7/9 | +| No reservoir column overdrawn (RNA) | 1/2 | 2/2 | 2/2 | 2/2 | 1/2 | 1/2 | +| Other RNA run-log checks | 32/32 | 32/32 | 32/32 | 32/32 | 33/33 | 33/33 | +| Traps tripped / lint violations | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
-The mechanical layers are close to solved: no model crashed the simulator or tried to game the grader, and the only physical-safety failure is Sonnet's reservoir overdraw. The signal is in the judge, and mostly in one item, `fidelity_to_paper`. +The mechanical layers are close to solved for the closed models: no model crashed the simulator or tried to game the grader, and their only physical-safety failure is Sonnet's reservoir overdraw. The open-weight models add the benchmark's only contamination failures. Beyond those, the signal is in the judge, and mostly in one item, `fidelity_to_paper`. ### 5.4 Error analysis -Every failed check and rubric item, with each judge vote and its evidence, is in [`REPORT.md`](results/runs/2026-10-07-openrouter/REPORT.md). They fall into five kinds: +Every failed check and rubric item, with each judge vote and its evidence, is in [`REPORT.md`](results/runs/2026-10-07-openrouter/REPORT.md). They fall into these kinds: -- **Inventing steps the paper does not describe** (all four models). On ecoli-hard, Sonnet, Opus and GPT all pipette-mix the competent cells after adding DNA, which the paper does not do and which harms fragile cells; Opus and GPT also mix after the SOC and choose their own lid temperatures. GPT adds a manual transfer of all 96 colony-PCR reactions to a second plate, with pauses (colony-hard, 0.50), and a 1 µL water "QC aliquot" step that leaves each Golden Gate well 1 µL over the paper's volume (golden-gate-hard). Sonnet and Fable add bead-slurry mixes in RNA extraction. -- **Not errors: recovering the whole eluate on RNA-hard, and a 10 µL colony PCR.** Sonnet, Opus and GPT recover all 100 µL on RNA-hard, and every model builds the paper's 10 µL colony-PCR reaction (5 µL master mix, 4 µL primer pair, 1 µL colony). Both follow the paper. Before the audit (Table 4, rows 7-8) the grader held them to the authors' 80 µL and the easy task's 20 µL; the affected scores are provisional (§6). On the easy RNA task, whose brief says 80 µL, over-recovering *is* an error. -- **Overdrawing a reservoir** (Sonnet, RNA easy, 0.30). Sonnet assigns six sample columns to four ethanol wells by `etoh[i % 4]`, so wells A9 and A10 each feed two columns: 8 × 500 µL × 2 columns × 2 washes = 16 mL from a 15 mL well. The new reservoir check fails it, which caps the reward at 0.30; the judge independently failed `robot_practice`. +- **Inventing steps the paper does not describe** (every model). On ecoli-hard, Sonnet, Opus and GPT all pipette-mix the competent cells after adding DNA, which the paper does not do and which harms fragile cells; Opus and GPT also mix after the SOC and choose their own lid temperatures. GPT adds a manual transfer of all 96 colony-PCR reactions to a second plate, with pauses (colony-hard, 0.50), and a 1 µL water "QC aliquot" step that leaves each Golden Gate well 1 µL over the paper's volume (golden-gate-hard). Sonnet and Fable add bead-slurry mixes in RNA extraction. Qwen fails `fidelity_to_paper` on all four paper-only tasks. +- **Not errors: recovering the whole eluate on RNA-hard, and a 10 µL colony PCR.** Sonnet, Opus and GPT recover all 100 µL on RNA-hard, and every model builds the paper's 10 µL colony-PCR reaction (5 µL master mix, 4 µL primer pair, 1 µL colony). Both follow the paper. Before the audit (Table 4, rows 7-8) the grader held them to the authors' 80 µL and the easy task's 20 µL; every affected trial has been re-judged. On the easy RNA task, whose brief says 80 µL, over-recovering *is* an error. +- **Overdrawing a reservoir** (Sonnet, Qwen and DeepSeek, RNA easy, 0.30 each). Sonnet assigns six sample columns to four ethanol wells by `etoh[i % 4]`, so wells A9 and A10 each feed two columns: 8 × 500 µL × 2 columns × 2 washes = 16 mL from a 15 mL well. DeepSeek's plan is the same and Qwen's overdraws one well the same way. The reservoir check fails it, which caps the reward at 0.30; the judge independently failed `robot_practice`. +- **One tip from the reaction back to the stock** (Qwen, both Golden Gate tasks; DeepSeek, AMPure and Golden-Gate-hard; 0.30 each). `transfer(..., new_tip='once', mix_after=(5, 15))` mixes in each reaction well and then returns the same tip to the enzyme-mix or bead stock for the next well, carrying every reaction into the stock. The contamination check and the judge's `tips_and_contamination` both fail it. No closed model did this. - **Wrong metadata** (Fable, RNA easy). Fable credits the RNA paper to the wrong authors, failing `fidelity_to_task`. +### 5.5 Local Docker or Modal, and how much one attempt moves + +We ran Qwen a second time on Modal (`-e modal -n 11`, every task at once) with the same agent, tasks and grader. +Times are from Harbor's `result.json` (`scripts/compare_jobs.py`); rewards are as graded at run time. + +
+ +**Table 8.** The same 11 tasks on two backends. + +| | Qwen, local Docker (2 at a time) | Qwen, Modal (11 at a time) | DeepSeek, Modal (11 at a time) | +|---|:---:|:---:|:---:| +| Wall clock, 11 tasks | 54.7 min | 31.9 min | 12.0 min | +| Median trial | 4.4 min | 7.8 min | 4.7 min | +| Median sandbox setup | 7 s | 1.5 min | 1.5 min | +| Median agent time | 3.0 min | 4.9 min | 1.8 min | +| Agent cost | $2.83 | $3.61 | $2.53 | +| Mean reward, 11 tasks | 0.728 | 0.805 | - | + +
+ +Modal is 1.7× faster for Qwen, and would be faster still without a single slow trial: run in parallel, the wall clock +is the slowest task (a 28-minute agent session on RNA extraction), and each sandbox spends about 1.5 minutes building +its image where local Docker reuses a cached one. The scores show the noise of one attempt: the mean moved by 0.08, and +four tasks changed by 0.08 to 0.70 with nothing changed but the run. Details: +[`results/repeats/2026-10-07-qwen-modal`](results/repeats/2026-10-07-qwen-modal/). + ## 6. Discussion and limitations -- **The paper is the hard part.** Easy tasks are nearly saturated; the paper-only tasks separate models, and the dominant failure there is inventing steps or quantities the paper does not support. GPT-6.1 Sol, perfect on easy tasks, fails `fidelity_to_paper` on all four paper-only ones (two provisional). The paper-only level is the one to grow, and every new task has to pass the audit first. -- **Refusals are not scored, but they are not free.** Two of four models refused standard molecular-cloning protocols. We exclude refused tasks from every score (§5.2), which makes the headline a comparison on 8 tasks, not 11; every new model can only shrink that set, so we also report each model's mean over all the tasks it answered, and its refusals. -- **Judge noise is real and now measured.** Three calls per protocol disagreed on 7 items across the run (each vote is in the grader record), and a single call had scored identical colony-PCR recipes differently. The majority vote absorbs this, but all signal still comes from one judge, a Claude model that also judges a non-Claude model; a second judge from another provider would show whether that matters. -- **Ground truth comes from the easy task.** Three of the four paper-only tasks reuse their easy task's ground truth. That is only fair where the paper fixes the same quantities; the audit now enforces it, and it is why colony-PCR-hard no longer grades its reaction volume deterministically. -- **Provisional scores.** Five scores, listed with reasons in [`PROVISIONAL.json`](results/runs/2026-10-07-openrouter/PROVISIONAL.json), were judged before the audit or with fewer than three votes: Sonnet and GPT on RNA-hard and colony-PCR-hard, and Sonnet on ecoli-hard (one vote). They are marked † on the leaderboard. The OpenRouter key ran out of credit before they could be re-judged; `python scripts/regrade_jobs.py --judge --all` does it for about $2. -- **Scope.** One robot (OT-2), one simulator, one agent harness, four models, one attempt per cell. Repeated attempts (`-k 3`) and non-Opentrons instruments (Hamilton, plate readers, imagers) are next. +- **The paper is the hard part.** Easy tasks are nearly saturated; the paper-only tasks separate models, and the dominant failure there is inventing steps or quantities the paper does not support. GPT-6.1 Sol, perfect on easy tasks, fails `fidelity_to_paper` on three of the four paper-only ones, and Qwen on all four. The paper-only level is the one to grow, and every new task has to pass the audit first. +- **Refusals are not scored, but they are not free.** Two of six models refused standard molecular-cloning protocols. We exclude refused tasks from every score (§5.2), which makes the headline a comparison on 8 tasks, not 11; every new model can only shrink that set, so we also report each model's mean over all the tasks it answered, and its refusals. +- **Judge noise is real and now measured.** Three calls per protocol disagreed on 9 of 380 items across the run (each vote is in the grader record), and a single call had scored identical colony-PCR recipes differently. The majority vote absorbs this, but all signal still comes from one judge, a Claude model that also judges a non-Claude model; a second judge from another provider would show whether that matters. +- **Ground truth comes from the easy task.** Three of the four paper-only tasks reuse their easy task's ground truth. That is only fair where the paper fixes the same quantities; the audit now enforces it, and it is why colony-PCR-hard checks its reaction's composition and the paper's volume range instead of the easy task's exact volumes. +- **One attempt is noisy.** Qwen, run a second time on Modal with everything else the same, moved from 0.728 to 0.805 over the 11 tasks, and four tasks changed score (RNA extraction 0.30 → 1.00, heat-shock-hard 0.75 → 0.30, Golden-Gate-hard 0.30 → 0.75). Gaps of under about 0.1 between models are not meaningful at pass@1 ([`results/repeats/`](results/repeats/2026-10-07-qwen-modal/)). +- **Scope.** One robot (OT-2), one simulator, one agent harness, six models, one attempt per cell. Repeated attempts (`-k 3`) and non-Opentrons instruments (Hamilton, plate readers, imagers) are next. ## 7. Reproducing diff --git a/docs/preprint/figures/grader_validation.png b/docs/preprint/figures/grader_validation.png index dfb58c0..ef01d49 100644 Binary files a/docs/preprint/figures/grader_validation.png and b/docs/preprint/figures/grader_validation.png differ diff --git a/docs/preprint/figures/run_means_cost.png b/docs/preprint/figures/run_means_cost.png index 57d706d..575c187 100644 Binary files a/docs/preprint/figures/run_means_cost.png and b/docs/preprint/figures/run_means_cost.png differ diff --git a/docs/preprint/figures/run_rewards.png b/docs/preprint/figures/run_rewards.png index f69aa21..56849e9 100644 Binary files a/docs/preprint/figures/run_rewards.png and b/docs/preprint/figures/run_rewards.png differ diff --git a/docs/preprint/figures/run_where_lost.png b/docs/preprint/figures/run_where_lost.png index d14e6eb..bb06fb1 100644 Binary files a/docs/preprint/figures/run_where_lost.png and b/docs/preprint/figures/run_where_lost.png differ diff --git a/docs/preprint/make_run_figures.py b/docs/preprint/make_run_figures.py index f093429..d3797f9 100644 --- a/docs/preprint/make_run_figures.py +++ b/docs/preprint/make_run_figures.py @@ -18,9 +18,11 @@ OUT = ROOT / "docs/preprint/figures" OUT.mkdir(parents=True, exist_ok=True) -ORDER = ["claude-sonnet-5.5", "claude-opus-5.5", "claude-fable-5.1", "gpt-6.1-sol"] -NAME = {"claude-sonnet-5.5": "Sonnet 5.5", "claude-opus-5.5": "Opus 5.5", "claude-fable-5.1": "Fable 5.1", "gpt-6.1-sol": "GPT-6.1 Sol"} -COL = {"claude-opus-5.5": "#2a78d6", "claude-sonnet-5.5": "#eb6834", "claude-fable-5.1": "#1baf7a", "gpt-6.1-sol": "#4a3aa7"} +ORDER = ["claude-sonnet-5.5", "claude-opus-5.5", "claude-fable-5.1", "gpt-6.1-sol", "qwen3.8-2.4t-a95b", "deepseek-v4-pro-0813"] +NAME = {"claude-sonnet-5.5": "Sonnet 5.5", "claude-opus-5.5": "Opus 5.5", "claude-fable-5.1": "Fable 5.1", "gpt-6.1-sol": "GPT-6.1 Sol", + "qwen3.8-2.4t-a95b": "Qwen3.8 2.4T", "deepseek-v4-pro-0813": "DeepSeek V4 Pro"} +COL = {"claude-opus-5.5": "#2a78d6", "claude-sonnet-5.5": "#eb6834", "claude-fable-5.1": "#1baf7a", "gpt-6.1-sol": "#4a3aa7", + "qwen3.8-2.4t-a95b": "#eda100", "deepseek-v4-pro-0813": "#e87ba4"} MODELS = [m for m in ORDER if (RUN / m).is_dir()] OFF = lambda j, width: (j - (len(MODELS) - 1) / 2) * width # bar offset of model j in a group INK, INK2, MUTED, GRID, REFUSE = "#0b0b0b", "#52514e", "#898781", "#e1e0d9", "#d9d7cf" @@ -79,7 +81,7 @@ def xgrid(ax): ax.xaxis.grid(True, color=GRID, lw=0.6); ax.set_axisbelow(True) # Figure 8: how the mean depends on refusals, and what it cost -fig, (a, b) = plt.subplots(1, 2, figsize=(9.2, 3.4), gridspec_kw={"width_ratios": [1.35, 1]}) +fig, (a, b) = plt.subplots(1, 2, figsize=(10.5, 3.8), gridspec_kw={"width_ratios": [1.6, 1]}) rules = [(f"The {len(common)} tasks\nall answered", lambda m: mean([reward(m, k) for k in common])), ("All tasks the\nmodel answered", lambda m: mean([reward(m, k) for k in answered[m]])), ("Easy tasks\nanswered", lambda m: mean([reward(m, k) for k in answered[m] if k in EASY])), @@ -90,32 +92,33 @@ def xgrid(ax): ax.xaxis.grid(True, color=GRID, lw=0.6); ax.set_axisbelow(True) xs = [i + OFF(j, w) for i in range(len(rules))] a.bar(xs, vals, w * .92, color=COL[m], label=NAME[m]) for x, v in zip(xs, vals): - a.text(x, v - .012, f"{v:.2f}", ha="center", va="top", fontsize=6.5, color="white", fontweight="bold") + a.text(x, v + .006, f"{v:.2f}".lstrip("0"), ha="center", va="bottom", fontsize=5.5, color=INK2, rotation=90) a.set_xticks(range(len(rules)), [r for r, _ in rules], fontsize=8) -a.set_ylim(.5, 1.0); a.set_ylabel("Mean reward"); ygrid(a) -a.legend(loc="upper center", bbox_to_anchor=(.5, -.2), ncol=len(MODELS), fontsize=7.5) +a.set_ylim(.5, 1.07); a.set_yticks([.5, .6, .7, .8, .9, 1.0]); a.set_ylabel("Mean reward"); ygrid(a) +a.legend(loc="upper center", bbox_to_anchor=(.5, -.2), ncol=(len(MODELS) + 1) // 2 if len(MODELS) > 4 else len(MODELS), fontsize=7.5) a.set_title("Refused tasks are not scored"); panel(a, "a") for m in MODELS: cost = sum(trials[(m, k)].get("cost_usd") or 0 for k in TASKS) v = rules[0][1](m) b.scatter(cost, v, s=90, color=COL[m], zorder=3) nref = sum(refused(m, k) for k in TASKS) - b.annotate(f"{NAME[m]}\n{nref} refused, not scored" if nref else NAME[m], (cost, v), xytext=(8, -4), textcoords="offset points", fontsize=8, color=INK2) + b.annotate(f"{NAME[m]}, {nref} refused" if nref else NAME[m], (cost, v), xytext=(8, -3), textcoords="offset points", fontsize=7.5, color=INK2) b.set_xlabel("Agent cost for the tasks it answered (USD)"); b.set_ylabel(f"Mean over the {len(common)} tasks all answered") costs = [sum(trials[(m, k)].get("cost_usd") or 0 for k in TASKS) for m in MODELS] vals = [rules[0][1](m) for m in MODELS] -b.set_xlim(0, max(costs) * 1.45); b.set_ylim(min(vals) - .05, min(1.02, max(vals) + .05)); ygrid(b); xgrid(b) +b.set_xlim(0, max(costs) * 1.75); b.set_ylim(min(vals) - .05, min(1.02, max(vals) + .05)); ygrid(b); xgrid(b) b.set_title("Cost against reward"); panel(b, "b") fig.tight_layout(); fig.savefig(OUT / "run_means_cost.png"); plt.close(fig) # Figure 9: where points were lost: deterministic risks vs judge rubric items -RISKS = [("simulator_ran", "Runs in the simulator"), ("end_state", "End-state volumes"), +RISKS = [("simulator_ran", "Runs in the simulator"), ("end_state", "End-state volumes\n(or paper-range mix)"), ("tip_before_aspirate", "Tip before pipetting"), ("no_overdispense", "No overdispense"), ("no_aspirate_from_empty_well", "No empty-well draw"), ("tip_dropped_at_end", "Tip dropped at end"), ("no_cross_contamination", "No cross-contamination"), ("rna", "RNA run-log checks")] def risk_of(name): base = name.split(":")[0].split("[")[0] # deck_labware is recorded only when it fails: none did + base = "end_state" if base == "composition" else base # colony-hard's plate: inputs, equal wells, paper's range return base if base in dict(RISKS) else "rna" passed, ran = collections.Counter(), collections.Counter() for (m, k), t in trials.items(): @@ -124,22 +127,18 @@ def risk_of(name): checks = checks.get("checks", []) for c in checks: passed[(m, risk_of(c["name"]))] += bool(c["pass"]); ran[(m, risk_of(c["name"]))] += 1 -fig, (a, b) = plt.subplots(1, 2, figsize=(10, 3.9), gridspec_kw={"width_ratios": [1, 1]}) -ys = range(len(RISKS)) -for j, m in enumerate(MODELS): - vals = [passed[(m, r)] / ran[(m, r)] if ran[(m, r)] else None for r, _ in RISKS] - for y, v, (r, _) in zip(ys, vals, RISKS): - if v is not None: - a.barh(y + OFF(j, .8 / len(MODELS)), v, .72 / len(MODELS), color=COL[m], label=NAME[m] if y == 0 else None) - -for y, (r, _) in zip(ys, RISKS): # one line per risk that any model missed: every model's passed/run - if any(ran[(m, r)] and passed[(m, r)] < ran[(m, r)] for m in MODELS): - a.text(1.02, y, "\n".join(f"{NAME[m]} {passed[(m, r)]}/{ran[(m, r)]}" for m in MODELS if ran[(m, r)]), - va="center", fontsize=5.8, color=INK2, linespacing=1.0) -a.set_yticks(list(ys), [l for _, l in RISKS], fontsize=8); a.invert_yaxis() -a.set_xlim(0, 1.3); a.set_xticks([0, .2, .4, .6, .8, 1]); a.set_xlabel("Share of checks passed"); xgrid(a) -a.set_title("Deterministic checks: almost nothing fails"); panel(a, "a") -a.legend(loc="lower left", fontsize=7.5, ncol=len(MODELS), bbox_to_anchor=(0, -.3)) +fig, (a, b) = plt.subplots(1, 2, figsize=(10.5, 3.9), gridspec_kw={"width_ratios": [1.1, 1]}) +for i, (r, _) in enumerate(RISKS): # one cell per risk and model: checks passed / run, red where any failed + for j, m in enumerate(MODELS): + n, ok = ran[(m, r)], passed[(m, r)] + face = "#f4f3ee" if not n else ("#2a78d6" if ok == n else "#b8321e") + a.add_patch(plt.Rectangle((j - .5, i - .5), 1, 1, facecolor=face, edgecolor="white", lw=1.5)) + a.text(j, i, f"{ok}/{n}" if n else "-", ha="center", va="center", fontsize=7, color="white" if n else MUTED) +a.set_xlim(-.5, len(MODELS) - .5); a.set_ylim(len(RISKS) - .5, -.5) +a.set_xticks(range(len(MODELS)), [NAME[m] for m in MODELS], rotation=30, ha="right", fontsize=7.5) +a.set_yticks(range(len(RISKS)), [l for _, l in RISKS], fontsize=7.5) +a.tick_params(length=0); [sp.set_visible(False) for sp in a.spines.values()] +a.set_title("Deterministic checks passed / run"); panel(a, "a") fails = collections.Counter() for (m, k), t in trials.items(): if m not in MODELS: @@ -157,8 +156,8 @@ def risk_of(name): b.set_yticks(range(len(ITEMS)), ITEMS, fontsize=8.5, family="monospace"); b.invert_yaxis() b.set_xlabel(f"Failed rubric items (all {len(MODELS)} models)"); b.xaxis.set_major_locator(matplotlib.ticker.MaxNLocator(integer=True)); xgrid(b) b.set_title("LLM judge: every lost point"); panel(b, "b") -b.text(.98, .03, f"{sum(fails.values())} failed items in all; every other item passed", - transform=b.transAxes, ha="right", fontsize=7.5, color=MUTED) +b.set_xlabel(f"Failed rubric items ({sum(fails.values())} in all; every other item passed)") +b.legend(loc="lower right", fontsize=7) fig.tight_layout(); fig.savefig(OUT / "run_where_lost.png"); plt.close(fig) @@ -178,9 +177,9 @@ def risk_of(name): ax.text(j, i, "n/a", ha="center", va="center", fontsize=6, color=MUTED) continue v = r.get("reward") or 0.0 - ctrl = a_.startswith("control") + ctrl = a_.startswith("control") or bool(r.get("valid")) # a valid variant (e.g. the paper's own volumes) must score 1 too face = ("#2a78d6" if v == 1 else "#b8321e") if ctrl else ("#b8321e" if v == 1 else SCORE(.5 + v * .4) if v > 0 else "#e7e5dc") - ax.add_patch(plt.Rectangle((j - .5, i - .5), 1, 1, facecolor=face, edgecolor="white", lw=1.5)) + ax.add_patch(plt.Rectangle((j - .5, i - .5), 1, 1, facecolor=face, edgecolor=INK if r.get("valid") else "white", lw=1.5)) ax.text(j, i, "1" if v == 1 else ("0" if v == 0 else f"{v:.2f}".lstrip("0")), ha="center", va="center", fontsize=6.5, color="white" if ctrl or v == 1 else INK) ax.axvline(2.5, color=INK, lw=1.2) @@ -188,11 +187,13 @@ def risk_of(name): ax.set_xticks(range(len(ATT)), [SHORT[a_] for a_ in ATT], rotation=55, ha="right", fontsize=7.5) ax.set_yticks(range(len(tasks)), tasks, fontsize=7.5) ax.tick_params(length=0); [s.set_visible(False) for s in ax.spines.values()] -holes = sum(1 for v in adv.values() for a_, r in v.items() if not a_.startswith("control") and r.get("reward") == 1.0) +holes = sum(1 for v in adv.values() for a_, r in v.items() if not a_.startswith("control") and not r.get("valid") and r.get("reward") == 1.0) +valid = sum(1 for v in adv.values() for r in v.values() if r.get("valid")) ctrl_ok = sum(1 for v in adv.values() for a_, r in v.items() if a_.startswith("control") and r.get("reward") == 1.0) ctrl_n = sum(1 for v in adv.values() for a_ in v if a_.startswith("control")) -ran = sum(1 for v in adv.values() for a_, r in v.items() if not a_.startswith("control") and "skipped" not in r) -ax.set_title(f"Controls: {ctrl_ok}/{ctrl_n} score 1.0 Attacks: {ran} run, {holes} scored 1.0", loc="left", fontsize=9.5) +ran = sum(1 for v in adv.values() for a_, r in v.items() if not a_.startswith("control") and "skipped" not in r and not r.get("valid")) +ax.set_title(f"Controls: {ctrl_ok}/{ctrl_n} score 1.0 Attacks: {ran} run, {holes} scored 1.0" + + (f" Valid variants: {valid} (score 1.0, outlined)" if valid else ""), loc="left", fontsize=9.5) ax.legend(handles=[Patch(color="#2a78d6", label="reference scores 1 (must)"), Patch(color="#e7e5dc", label="attack scores 0"), Patch(color=SCORE(.6), label="attack: partial credit"), Patch(color="#b8321e", label="hole (attack scores 1)")], loc="upper left", bbox_to_anchor=(0, -.42), ncol=4, fontsize=7.5) diff --git a/docs/preprint/preprint.html b/docs/preprint/preprint.html index 6acb2d1..63cdb9b 100644 --- a/docs/preprint/preprint.html +++ b/docs/preprint/preprint.html @@ -112,22 +112,24 @@

Abstract

simulator, deterministic checks of the simulated run against a ground-truth protocol representation, and a pass/fail rubric scored by a language-model judge as the majority of three calls. We validate the -verifier from both sides: all 11 reference solutions pass, 152 broken or +verifier from both sides: all 11 reference solutions pass, 151 broken or cheating protocols of 18 kinds all fail (none scores above 0.47), a -close reading of the model runs exposed six grader faults, and an audit -requiring every quantity a paper-only task grades to be in its paper or -brief found two more; all eight are fixed. Running one agent harness -(Claude Code) with four models through one API, and scoring only the -tasks a model answered, Claude Opus 5.5 scores 1.000 on the eight tasks -every model answered, Fable 5.1 0.969, GPT-6.1 Sol 0.898 and Sonnet 5.5 -0.842. Opus and Fable refused five standard molecular-cloning prompts -between them; a refusal is the provider’s safety policy, not a protocol, -so these are reported but not scored. No model failed the simulator or a -trap, and only one broke a physical safety rule. Easy tasks are nearly -solved; the paper-only tasks separate the models, and their commonest -failure is inventing steps and quantities that the paper does not -support. Five of the 44 scores are provisional until a re-judge with the -audited verifier.

+close reading of the model runs exposed eight grader faults, and an +audit requiring every quantity a paper-only task grades to be in its +paper or brief found two more; all ten are fixed. Running one agent +harness (Claude Code) with six models through one API, and scoring only +the tasks a model answered, Claude Opus 5.5 scores 1.000 on the eight +tasks every model answered, Fable 5.1 0.969, GPT-6.1 Sol 0.938, Sonnet +5.5 0.881, Qwen3.8-2.4T-A95B 0.832 and DeepSeek V4 Pro 0.755. Opus and +Fable refused five standard molecular-cloning prompts between them; a +refusal is the provider’s safety policy, not a protocol, so these are +reported but not scored. No model failed the simulator or a trap. The +physical-safety failures are three reservoir overdraws and, in the two +open-weight models only, four cross-contaminations. Easy tasks are +nearly solved; the paper-only tasks separate the models, and their +commonest failure is inventing steps and quantities that the paper does +not support. A repeat run of one model moved its mean by 0.08 and one +task by 0.70.

1 Introduction

Automating a published protocol on a liquid-handling robot is a translation problem with two failure modes. The first is mechanical: @@ -157,13 +159,15 @@

1 Introduction

against a ground-truth protocol representation, a 75/25 core/task rubric scored by a three-vote judge, and ten reward-hacking traps (Section 4). -
  • A validation of that verifier against reference solutions and 152 +
  • A validation of that verifier against reference solutions and 151 adversarial protocols, an audit that every quantity a paper-only task -grades is in its paper or brief, and eight grader faults found by -reading model runs, each fixed with a regression test (Section 5).
    • -
  • A comparison of four frontier models from two providers in one agent -harness, broken down by task, risk and rubric item, with safety refusals -reported separately from capability (Sections 6 and 7).
    • +grades is in its paper or brief, and ten grader faults found by reading +model runs, each fixed with a regression test (Section 5). +
  • A comparison of six models from four providers, two of them +open-weight, in one agent harness, broken down by task, risk and rubric +item, with safety refusals reported separately from capability, and a +repeat run that measures single-attempt noise and a parallel cloud +backend (Sections 6 and 7).

    • Work on language models for wet-lab protocols spans four questions: @@ -533,24 +537,27 @@

    5 Does the verifier grade simulated run and the loaded labware unchanged is reported as not applicable rather than scored.

    -Figure 2. Grader validation. Deterministic reward (judge off) for the reference solution at three API levels (left of the line, must be 1) and 152 attacks on nine tasks (right, must be below 1). All 27 controls score 1 and no attack does; the best-scoring attack gets 0.47, and every attack on the grader scores 0. +Figure 2. Grader validation. Deterministic reward (judge off) for the reference solution at three API levels (left of the line, must be 1) and 151 attacks on nine tasks (right, must be below 1). All 27 controls score 1 and no attack does; the best-scoring attack gets 0.47, and every attack on the grader scores 0. Outlined: halving every volume on colony-PCR-hard gives the paper’s own 10 µL reaction, a valid protocol that must score 1. +grader scores 0. Outlined: halving every volume on colony-PCR-hard gives +the paper’s own 10 µL reaction, a valid protocol that must score +1.

    Model runs taught the verifier. We read every failed check and rubric item from the runs in Section 7 and confirmed each -against the protocol. Six faults turned up (Table 3). A second check, +against the protocol. Eight faults turned up (Table 3), two of them in +the open-weight models’ runs. A second check, tests/test_hard_tasks.py, requires every volume a paper-only task grades deterministically to be in its paper or brief, or to follow from quoted facts by arithmetic the test spells out; it found two more. Each fix has a regression test.

    Table 3. Grader faults found by reading the model -runs (rows 1-6) and by the paper-only audit (rows 7-8).

    +runs (rows 1-6 and 9-10) and by the paper-only audit (rows 7-8).

    @@ -610,7 +617,21 @@

    5 Does the verifier grade

    - + + + + + + + + + + +
    Colony-PCR-hard graded the easy task’s 20 µL reaction; the paper’s is 10 µL, with no primer concentration on the deck Every model “failed” itThe reaction volume is left to the rubricNo exact volumes: every well must get all three inputs, alike, 10-25 +µL
    The lint gate refused Opentrons’ documented +pipette.tip_racks = [...]DeepSeek, a correct protocol scored 0Documented pipette settings allowed; patching classes or modules +still refused
    Mixing a source well before drawing from it (resuspending a colony) +was flagged as contaminationDeepSeek, colony-hard (0.30, really 1.00)Only mixing in a destination makes a tip carry
    @@ -623,41 +644,48 @@

    5 Does the verifier grade template”, the brief’s stock concentrations and the polymerase maker’s standard recipe. RNA-hard grades only the paper’s 40/250/250 µL, two 500 µL washes, 4 min drying, 100 µL elution and the brief’s 4 °C plate. -Colony-PCR-hard keeps its deck, tip and contamination checks and leaves -the reaction to the rubric. Whether the level is harder is a -weaker claim: its briefs list no steps and few volumes, but each model -has only 2-4 paper-only trials, so we call it paper-only rather than -hard.

    +Colony-PCR-hard keeps its deck, tip and contamination checks and, +instead of exact volumes, requires all three inputs in every well, +alike, and a reaction of 10-25 µL, from the paper’s “9 μL of this master +mix” plus 1 µL colony up to a standard Q5 reaction. Whether the level is +harder is a weaker claim: its briefs list no steps and few +volumes, but each model has only 2-4 paper-only trials, so we call it +paper-only rather than hard.

    6 Experimental setup

    We ran Claude Code 2.1.288 as the agent for every model through OpenRouter’s Anthropic-compatible API, with Harbor 0.23.0 on Docker, 2 CPUs and 2.5 GB per sandbox, one attempt per task and model (pass@1). -The models were Claude Sonnet 5.5, Opus 5.5 and Fable 5.1 (Anthropic) -and GPT-6.1 Sol (OpenAI). Harbor pins every model alias and sub-agent of -the harness to the model under test, and we checked that every -transcript contains only that model’s responses. The judge was Claude -Sonnet 5.5 throughout. The Claude trials were regraded from their saved -protocols with the final verifier; the GPT trials were graded by it -directly. Every graded protocol, check, judge vote and cost is released +The models were Claude Sonnet 5.5, Opus 5.5 and Fable 5.1 (Anthropic), +GPT-6.1 Sol (OpenAI), and the open-weight Qwen3.8-2.4T-A95B (Alibaba) +and DeepSeek V4 Pro (DeepSeek). DeepSeek ran on Modal with all 11 tasks +in parallel; the others ran locally, two at a time, and Qwen was +repeated on Modal. Harbor pins every model alias and sub-agent of the +harness to the model under test, and we checked that every transcript +contains only that model’s responses. The judge was Claude Sonnet 5.5 +throughout, and every trial is graded by the final verifier, earlier +ones regraded from their saved protocols. Agent cost is computed from +token counts at OpenRouter prices, since the harness prices every model +as Claude. Every graded protocol, check, judge vote and cost is released with the benchmark.

    7 Results

    -Figure 3. Reward on all 11 tasks, one attempt per task and model. Hatched cells are refusals. Above the line, easy tasks; below, paper-only tasks. +Figure 3. Reward on all 11 tasks, one attempt per task and model. Hatched cells are refusals. Above the line, easy tasks; below, paper-only tasks.

    Table 4. Summary. Refused tasks are not scored; the -first row is the headline. † Includes provisional scores (Section -8).

    +first row is the headline.

    -----+++++++ @@ -666,22 +694,28 @@

    7 Results

    + + - + - + + + - + - + + + @@ -689,20 +723,35 @@

    7 Results

    + + - + - + + + - + + + + + + + + + + + + @@ -710,85 +759,105 @@

    7 Results

    + + - + + +
    Opus 5.5 Fable 5.1 GPT-6.1 SolQwen3.8-2.4TDeepSeek V4 Pro
    Mean over the 8 tasks all answered0.8420.881 1.000 0.9690.8980.9380.8320.755
    Mean over all tasks answered0.863 (11)0.891 (11) 0.972 (9) 0.969 (8)0.881 (11)0.909 (11)0.728 (11)0.735 (11)
    Easy tasks, answered 1.000 0.958 1.0000.8000.800
    Paper-only tasks, answered0.797†0.875 0.917 1.0000.672†0.7500.6010.622
    fidelity_to_paper passed1/42/4 2/3 2/21/4 0/42/4
    Critical failures (capped at 0.30)100033
    Refused, not scored 2 3 000
    Agent cost (USD) 1.25 3.52 6.214.821.492.832.53
    -

    Easy tasks are nearly solved. Six of the seven easy -tasks gave every model that answered them full marks, and GPT-6.1 Sol -and Opus were perfect on every easy task they answered. The models -separate on RNA extraction and on the hard tasks, where 8 of 13 answered -trials failed fidelity_to_paper.

    +

    Easy tasks are nearly solved. Four of the seven easy +tasks gave every model full marks, and GPT-6.1 Sol and Opus were perfect +on every easy task they answered. The models separate on RNA extraction +and on the hard tasks, where 12 of 21 answered trials failed +fidelity_to_paper. The two open-weight models trail on both +levels.

    Refused tasks are not scored. Anthropic’s biosecurity safeguard declined both Golden Gate prompts for Opus and Fable and the paper-only E. coli transformation prompt for Fable. These are teaching-lab procedures, so the refusals are false positives; we report them as they happened and did not rephrase prompts. A refusal is the provider’s policy, not a protocol, and says nothing about a model’s -ability to write one, so we compare models on the eight tasks all four +ability to write one, so we compare models on the eight tasks all six answered and report refusals separately (Figure 4). Opus leads there, -with Fable, GPT and Sonnet a few judge items behind; Sonnet is the -cheapest, and Sonnet and GPT are the only models that answered every -task. ABC-Bench reports refusals on its dual-use screening-evasion task, -where the tested Anthropic and OpenAI frontier models refused every -sample (A. B. Liu et al. +with Fable, GPT and Sonnet a few judge items behind, then Qwen and +DeepSeek; Sonnet is the cheapest, and four of the six models answered +every task. ABC-Bench reports refusals on its dual-use screening-evasion +task, where the tested Anthropic and OpenAI frontier models refused +every sample (A. B. Liu et al. 2026); ours fall on benign tasks.

    -Figure 4. (a) Mean reward over the eight tasks all four models answered (the headline), over every task each model answered, and over the easy and paper-only tasks it answered. (b) Agent cost for the tasks each model answered against its headline score. +Figure 4. (a) Mean reward over the eight tasks all six models answered (the headline), over every task each model answered, and over the easy and paper-only tasks it answered. (b) Agent cost for the tasks each model answered against its headline score. +reward over the eight tasks all six models answered (the headline), over +every task each model answered, and over the easy and paper-only tasks +it answered. (b) Agent cost for the tasks each model answered against +its headline score.
    -

    The mechanical layers are solved; the signal is in the +

    The mechanical layers are nearly solved; the signal is in the judge. No model crashed the simulator, tripped a trap or broke -the lint gate, and the only physical-safety failure is Sonnet’s -reservoir overdraw (Figure 5). End-state misses are the colony-PCR-hard -plate for every model, which built the paper’s 10 µL reaction where the -ground truth fixes 20 µL, and GPT’s Golden-Gate-hard plate.

    +the lint gate (Figure 5). The closed models’ only physical-safety +failure is Sonnet’s reservoir overdraw; Qwen and DeepSeek overdraw the +same reservoir and add the run’s only cross-contaminations. End-state +misses are GPT’s and Qwen’s Golden-Gate-hard plates.

    -Figure 5. (a) Deterministic checks passed per risk and model. (b) Every rubric item the judge failed: 14 failures in 39 judged trials; tips_and_contamination passed in all 39. +Figure 5. (a) Deterministic checks passed / run per risk and model; red where any failed. (b) Every rubric item the judge failed: 28 failures in 61 judged trials. +Deterministic checks passed / run per risk and model; red where any +failed. (b) Every rubric item the judge failed: 28 failures in 61 judged +trials.
    -

    Error analysis. Three kinds of error account for +

    Error analysis. Four kinds of error account for every lost point that the audited verifier upholds. Inventing steps -the paper does not describe (all four models): Sonnet, Opus and GPT -pipette-mix competent cells after adding DNA on ecoli-hard, which harms -fragile cells; GPT adds a manual transfer of all 96 PCR reactions to a -second plate and a 1 µL water “QC aliquot” in Golden Gate. -Overdrawing a reservoir: Sonnet maps six sample columns onto -four ethanol wells, drawing 16 mL from a 15 mL well. Wrong -metadata: Fable credits the RNA paper to the wrong authors. Two -choices that looked like errors are not: recovering the whole 100 µL -eluate on RNA-hard and building the paper’s 10 µL colony-PCR reaction -both follow the paper; the scores that held them to the authors’ 80 µL -and the easy task’s 20 µL are provisional.

    +the paper does not describe (every model; Qwen fails +fidelity_to_paper on all four paper-only tasks): Sonnet, +Opus and GPT pipette-mix competent cells after adding DNA on ecoli-hard, +which harms fragile cells; GPT adds a manual transfer of all 96 PCR +reactions to a second plate and a 1 µL water “QC aliquot” in Golden +Gate. Overdrawing a reservoir (Sonnet, Qwen, DeepSeek): six +sample columns mapped onto four ethanol wells draw 16 mL from a 15 mL +well. One tip from the reaction back to the stock (Qwen on both +Golden Gate tasks, DeepSeek on AMPure and Golden-Gate-hard): +transfer(..., new_tip='once', mix_after=...) mixes in each +reaction and returns the same tip to the enzyme or bead stock, carrying +every reaction into it. Wrong metadata: Fable credits the RNA +paper to the wrong authors. Two choices that looked like errors are not: +recovering the whole 100 µL eluate on RNA-hard and building the paper’s +10 µL colony-PCR reaction both follow the paper, and every trial graded +before the audit has been re-judged.

    +

    One attempt is noisy, and the cloud is faster. Qwen +run a second time on Modal, everything else unchanged, scored 0.805 +against 0.728 locally; four tasks changed, by up to 0.70 (RNA extraction +0.30 to 1.00). Gaps of under about 0.1 between models are within this +noise. With all 11 tasks in parallel Modal finished in 31.9 minutes +against 54.7 for local Docker two at a time, despite about 1.5 minutes +of image build per sandbox; the wall clock is then the slowest trial, +and DeepSeek, without a slow trial, finished all 11 in 12 minutes.

    8 Discussion and limitations

    The paper is the hard part. Easy tasks saturate; the paper-only tasks separate models, and their dominant failure is inventing steps or quantities. GPT-6.1 Sol, perfect on easy tasks, -failed fidelity_to_paper on all four paper-only ones (two -provisional). The paper-only level is the one to grow, and each new task -must pass the audit.

    -

    Verifiers need adversaries. Eight grader faults +failed fidelity_to_paper on three of the four paper-only +ones, and Qwen on all four. The paper-only level is the one to grow, and +each new task must pass the audit.

    +

    Verifiers need adversaries. Ten grader faults survived reference solutions, unit tests and an adversarial suite, and surfaced only when we read what real models wrote or audited the ground -truth against the paper. One was a false negative that capped a correct -protocol at 0.30. Benchmarks that grade generated laboratory code should -publish their graded artefacts so faults like these can be found.

    +truth against the paper. Three were false negatives that capped or +zeroed a correct protocol, two of them found only when new models wrote +code in styles the earlier models had not. Benchmarks that grade +generated laboratory code should publish their graded artefacts so +faults like these can be found.

    Judge noise is real. Three calls per protocol -disagreed on seven items across the run, and single calls had scored +disagreed on 9 of 380 items across the run, and single calls had scored identical recipes differently. Majority voting absorbs this, but all signal still comes from one judge, a Claude model that also judges a non-Claude model. A second judge from another provider, as AEGIS does @@ -797,17 +866,12 @@

    8 Discussion and limitations

    Ground truth comes from the easy task. Three of the four paper-only tasks reuse their easy task’s ground truth, which is fair only where the paper fixes the same quantities. The audit now -enforces that.

    -

    Provisional scores. Five scores were judged before -the audit or with fewer than three votes: Sonnet and GPT on RNA-hard and -colony-PCR-hard, and Sonnet on heat-shock-hard. They are marked † in -Table 4 and on the leaderboard, with reasons in the released results. -API credit ran out before they could be re-judged; an open-weight model -(Qwen3.8) is the next run.

    +enforces that; where the paper fixes less, as in colony-PCR-hard, the +checks fall back to what it does fix.

    Scope. One robot (OT-2), one simulator, one agent -harness, four models, one attempt per cell. Repeated attempts, wet-lab -execution of the generated protocols and non-Opentrons instruments are -future work.

    +harness, six models, one attempt per cell (one model repeated). Repeated +attempts for every model, wet-lab execution of the generated protocols +and non-Opentrons instruments are future work.

    9 Conclusion

    Text2WetLab measures whether agents turn lab protocols and papers into robot code that does what the paper describes, not merely code that @@ -816,8 +880,8 @@

    9 Conclusion

    quantities of their own. Safety refusals on routine molecular biology are common enough that a benchmark has to keep them out of its scores and report them on their own. And the grader is itself an artefact to -validate: ours needed eight fixes, found only by reading real model -output and auditing the ground truth against the papers.

    +validate: ours needed ten fixes, found only by reading real model output +and auditing the ground truth against the papers.

    Data and code availability

    Tasks, verifier, reference solutions, adversarial suite, every graded protocol and grader record, and the scripts that produce every table and diff --git a/eval/protocol_lint.py b/eval/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/eval/protocol_lint.py +++ b/eval/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/eval/spec_check.py b/eval/spec_check.py index 7a1e805..4e256e3 100644 --- a/eval/spec_check.py +++ b/eval/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/manuscript/main.md b/manuscript/main.md index e2acfa9..9b21000 100644 --- a/manuscript/main.md +++ b/manuscript/main.md @@ -19,16 +19,17 @@ exact steps; *paper-only* ("hard") tasks give only a goal, a fixed deck and the protocols whose authors also released their robot code, which gives an independent reference for each. A layered verifier scores the agent's Opentrons OT-2 protocol with a lint gate, ten reward-hacking traps, the Opentrons simulator, deterministic checks of the simulated run against a ground-truth protocol representation, and a pass/fail rubric scored by a language-model judge as the majority of -three calls. We validate the verifier from both sides: all 11 reference solutions pass, 152 broken or cheating -protocols of 18 kinds all fail (none scores above 0.47), a close reading of the model runs exposed six grader +three calls. We validate the verifier from both sides: all 11 reference solutions pass, 151 broken or cheating +protocols of 18 kinds all fail (none scores above 0.47), a close reading of the model runs exposed eight grader faults, and an audit requiring every quantity a paper-only task grades to be in its paper or brief found two more; -all eight are fixed. Running one agent harness (Claude Code) with four models through one API, and scoring only the +all ten are fixed. Running one agent harness (Claude Code) with six models through one API, and scoring only the tasks a model answered, Claude Opus 5.5 scores 1.000 on the eight tasks every model answered, Fable 5.1 0.969, GPT-6.1 -Sol 0.898 and Sonnet 5.5 0.842. Opus and Fable refused five standard molecular-cloning prompts between them; a refusal -is the provider's safety policy, not a protocol, so these are reported but not scored. No model failed the simulator or a trap, and only one broke a physical -safety rule. Easy tasks are nearly solved; the paper-only tasks separate the models, and their commonest failure is -inventing steps and quantities that the paper does not support. Five of the 44 scores are provisional until a -re-judge with the audited verifier. +Sol 0.938, Sonnet 5.5 0.881, Qwen3.8-2.4T-A95B 0.832 and DeepSeek V4 Pro 0.755. Opus and Fable refused five standard +molecular-cloning prompts between them; a refusal is the provider's safety policy, not a protocol, so these are +reported but not scored. No model failed the simulator or a trap. The physical-safety failures are three reservoir +overdraws and, in the two open-weight models only, four cross-contaminations. Easy tasks are nearly solved; the +paper-only tasks separate the models, and their commonest failure is inventing steps and quantities that the paper +does not support. A repeat run of one model moved its mean by 0.08 and one task by 0.70. ## 1 Introduction @@ -49,11 +50,12 @@ quantity checked in the simulated run against a ground truth, and whether the gr source paper, which it must read (Section 3). 2. A layered verifier: deterministic checks of the simulated run against a ground-truth protocol representation, a 75/25 core/task rubric scored by a three-vote judge, and ten reward-hacking traps (Section 4). -3. A validation of that verifier against reference solutions and 152 adversarial protocols, an audit that every - quantity a paper-only task grades is in its paper or brief, and eight grader faults +3. A validation of that verifier against reference solutions and 151 adversarial protocols, an audit that every + quantity a paper-only task grades is in its paper or brief, and ten grader faults found by reading model runs, each fixed with a regression test (Section 5). -4. A comparison of four frontier models from two providers in one agent harness, broken down by task, risk and rubric - item, with safety refusals reported separately from capability (Sections 6 and 7). +4. A comparison of six models from four providers, two of them open-weight, in one agent harness, broken down by task, + risk and rubric item, with safety refusals reported separately from capability, and a repeat run that measures + single-attempt noise and a parallel cloud backend (Sections 6 and 7). ## 2 Related work @@ -226,17 +228,18 @@ left on) and attacks on the grader (an empty protocol, comments that only claim and the loaded labware unchanged is reported as not applicable rather than scored. ![**Figure 2.** Grader validation. Deterministic reward (judge off) for the reference solution at three API levels -(left of the line, must be 1) and 152 attacks on nine tasks (right, must be below 1). All 27 controls score 1 and no -attack does; the best-scoring attack gets 0.47, and every attack on the grader scores 0.](../docs/preprint/figures/grader_validation.png) +(left of the line, must be 1) and 151 attacks on nine tasks (right, must be below 1). All 27 controls score 1 and no +attack does; the best-scoring attack gets 0.47, and every attack on the grader scores 0. Outlined: halving every volume +on colony-PCR-hard gives the paper's own 10 µL reaction, a valid protocol that must score 1.](../docs/preprint/figures/grader_validation.png) **Model runs taught the verifier.** We read every failed check and rubric item from the runs in Section 7 and confirmed -each against the protocol. Six faults turned up (Table 3). A second check, `tests/test_hard_tasks.py`, requires every +each against the protocol. Eight faults turned up (Table 3), two of them in the open-weight models' runs. A second check, `tests/test_hard_tasks.py`, requires every volume a paper-only task grades deterministically to be in its paper or brief, or to follow from quoted facts by arithmetic the test spells out; it found two more. Each fix has a regression test.

    -**Table 3.** Grader faults found by reading the model runs (rows 1-6) and by the paper-only audit (rows 7-8). +**Table 3.** Grader faults found by reading the model runs (rows 1-6 and 9-10) and by the paper-only audit (rows 7-8). | Fault | Found in | Fix | |---|---|---| @@ -247,7 +250,9 @@ arithmetic the test spells out; it found two more. Each fix has a regression tes | One judge call scored identical recipes differently | Colony-PCR-hard | Three calls, majority per item; primer volume left to the agent, as the deck gives no concentration | | End-state details truncated to 160 characters | Every colony-hard report | Full expected and measured values | | RNA-hard's rubric required 80 µL recovered; the paper says only "collect the supernatant" | Sonnet, GPT marked down | Any full recovery passes on the paper-only task | -| Colony-PCR-hard graded the easy task's 20 µL reaction; the paper's is 10 µL, with no primer concentration on the deck | Every model "failed" it | The reaction volume is left to the rubric | +| Colony-PCR-hard graded the easy task's 20 µL reaction; the paper's is 10 µL, with no primer concentration on the deck | Every model "failed" it | No exact volumes: every well must get all three inputs, alike, 10-25 µL | +| The lint gate refused Opentrons' documented `pipette.tip_racks = [...]` | DeepSeek, a correct protocol scored 0 | Documented pipette settings allowed; patching classes or modules still refused | +| Mixing a source well before drawing from it (resuspending a colony) was flagged as contamination | DeepSeek, colony-hard (0.30, really 1.00) | Only mixing in a destination makes a tip carry |
    @@ -256,18 +261,21 @@ its own ground truth from the APEX paper: temperatures and times in the text, vo hard's master-mix volumes follow from the paper's "25 µL volumes … 0.1 µM primers and 0.5 ng … template", the brief's stock concentrations and the polymerase maker's standard recipe. RNA-hard grades only the paper's 40/250/250 µL, two 500 µL washes, 4 min drying, 100 µL elution and the brief's 4 °C plate. Colony-PCR-hard keeps its deck, tip and -contamination checks and leaves the reaction to the rubric. Whether the level is *harder* is a weaker claim: its briefs +contamination checks and, instead of exact volumes, requires all three inputs in every well, alike, and a reaction of +10-25 µL, from the paper's "9 μL of this master mix" plus 1 µL colony up to a standard Q5 reaction. Whether the level is *harder* is a weaker claim: its briefs list no steps and few volumes, but each model has only 2-4 paper-only trials, so we call it paper-only rather than hard. ## 6 Experimental setup We ran Claude Code 2.1.288 as the agent for every model through OpenRouter's Anthropic-compatible API, with Harbor 0.23.0 on Docker, 2 CPUs and 2.5 GB per sandbox, one attempt per task and model (pass@1). The models were Claude Sonnet -5.5, Opus 5.5 and Fable 5.1 (Anthropic) and GPT-6.1 Sol (OpenAI). Harbor pins every model alias and sub-agent of the -harness to the model under test, and we checked that every transcript contains only that model's responses. The judge -was Claude Sonnet 5.5 throughout. The Claude trials were regraded from their saved protocols with the final verifier; -the GPT trials were graded by it directly. Every graded protocol, check, judge vote and cost is released with the -benchmark. +5.5, Opus 5.5 and Fable 5.1 (Anthropic), GPT-6.1 Sol (OpenAI), and the open-weight Qwen3.8-2.4T-A95B (Alibaba) and +DeepSeek V4 Pro (DeepSeek). DeepSeek ran on Modal with all 11 tasks in parallel; the others ran locally, two at a time, +and Qwen was repeated on Modal. Harbor pins every model alias and sub-agent of the harness to the model under test, and +we checked that every transcript contains only that model's responses. The judge was Claude Sonnet 5.5 throughout, and +every trial is graded by the final verifier, earlier ones regraded from their saved protocols. Agent cost is computed +from token counts at OpenRouter prices, since the harness prices every model as Claude. Every graded protocol, check, +judge vote and cost is released with the benchmark. ## 7 Results @@ -276,78 +284,82 @@ tasks; below, paper-only tasks.](../docs/preprint/figures/run_rewards.png)
    -**Table 4.** Summary. Refused tasks are not scored; the first row is the headline. † Includes provisional scores (Section 8). +**Table 4.** Summary. Refused tasks are not scored; the first row is the headline. -| | Sonnet 5.5 | Opus 5.5 | Fable 5.1 | GPT-6.1 Sol | -|---|:---:|:---:|:---:|:---:| -| **Mean over the 8 tasks all answered** | 0.842 | **1.000** | 0.969 | 0.898 | -| Mean over all tasks answered | 0.863 (11) | **0.972** (9) | 0.969 (8) | 0.881 (11) | -| Easy tasks, answered | 0.900 | **1.000** | 0.958 | **1.000** | -| Paper-only tasks, answered | 0.797† | 0.917 | **1.000** | 0.672† | -| `fidelity_to_paper` passed | 1/4 | 2/3 | 2/2 | 0/4 | -| Refused, not scored | 0 | 2 | 3 | 0 | -| Agent cost (USD) | **1.25** | 3.52 | 6.21 | 4.82 | +| | Sonnet 5.5 | Opus 5.5 | Fable 5.1 | GPT-6.1 Sol | Qwen3.8-2.4T | DeepSeek V4 Pro | +|---|:---:|:---:|:---:|:---:|:---:|:---:| +| **Mean over the 8 tasks all answered** | 0.881 | **1.000** | 0.969 | 0.938 | 0.832 | 0.755 | +| Mean over all tasks answered | 0.891 (11) | **0.972** (9) | 0.969 (8) | 0.909 (11) | 0.728 (11) | 0.735 (11) | +| Easy tasks, answered | 0.900 | **1.000** | 0.958 | **1.000** | 0.800 | 0.800 | +| Paper-only tasks, answered | 0.875 | 0.917 | **1.000** | 0.750 | 0.601 | 0.622 | +| `fidelity_to_paper` passed | 2/4 | 2/3 | 2/2 | 1/4 | 0/4 | 2/4 | +| Critical failures (capped at 0.30) | 1 | 0 | 0 | 0 | 3 | 3 | +| Refused, not scored | 0 | 2 | 3 | 0 | 0 | 0 | +| Agent cost (USD) | **1.25** | 3.52 | 6.21 | 1.49 | 2.83 | 2.53 |
    -**Easy tasks are nearly solved.** Six of the seven easy tasks gave every model that answered them full marks, and -GPT-6.1 Sol and Opus were perfect on every easy task they answered. The models separate on RNA extraction and on the hard tasks, -where 8 of 13 answered trials failed `fidelity_to_paper`. +**Easy tasks are nearly solved.** Four of the seven easy tasks gave every model full marks, and GPT-6.1 Sol and Opus +were perfect on every easy task they answered. The models separate on RNA extraction and on the hard tasks, where 12 of +21 answered trials failed `fidelity_to_paper`. The two open-weight models trail on both levels. **Refused tasks are not scored.** Anthropic's biosecurity safeguard declined both Golden Gate prompts for Opus and Fable and the paper-only E. coli transformation prompt for Fable. These are teaching-lab procedures, so the refusals are false positives; we report them as they happened and did not rephrase prompts. A refusal is the provider's policy, not -a protocol, and says nothing about a model's ability to write one, so we compare models on the eight tasks all four +a protocol, and says nothing about a model's ability to write one, so we compare models on the eight tasks all six answered and report refusals separately (Figure 4). Opus leads there, with Fable, GPT and Sonnet a few judge items -behind; Sonnet is the cheapest, and Sonnet and GPT are the only models that answered every task. ABC-Bench reports refusals on +behind, then Qwen and DeepSeek; Sonnet is the cheapest, and four of the six models answered every task. ABC-Bench reports refusals on its dual-use screening-evasion task, where the tested Anthropic and OpenAI frontier models refused every sample [@liu2026abcbench]; ours fall on benign tasks. -![**Figure 4.** (a) Mean reward over the eight tasks all four models answered (the headline), over every task each +![**Figure 4.** (a) Mean reward over the eight tasks all six models answered (the headline), over every task each model answered, and over the easy and paper-only tasks it answered. (b) Agent cost for the tasks each model answered against its headline score.](../docs/preprint/figures/run_means_cost.png) -**The mechanical layers are solved; the signal is in the judge.** No model crashed the simulator, tripped a trap or -broke the lint gate, and the only physical-safety failure is Sonnet's reservoir overdraw (Figure 5). End-state misses are -the colony-PCR-hard plate for every model, which built the paper's 10 µL reaction where the ground truth fixes 20 µL, -and GPT's Golden-Gate-hard plate. +**The mechanical layers are nearly solved; the signal is in the judge.** No model crashed the simulator, tripped a +trap or broke the lint gate (Figure 5). The closed models' only physical-safety failure is Sonnet's reservoir overdraw; +Qwen and DeepSeek overdraw the same reservoir and add the run's only cross-contaminations. End-state misses are GPT's +and Qwen's Golden-Gate-hard plates. -![**Figure 5.** (a) Deterministic checks passed per risk and model. (b) Every rubric item the judge failed: 14 failures -in 39 judged trials; `tips_and_contamination` passed in all 39.](../docs/preprint/figures/run_where_lost.png) +![**Figure 5.** (a) Deterministic checks passed / run per risk and model; red where any failed. (b) Every rubric item +the judge failed: 28 failures in 61 judged trials.](../docs/preprint/figures/run_where_lost.png) -**Error analysis.** Three kinds of error account for every lost point that the audited verifier upholds. *Inventing steps the paper does not describe* -(all four models): Sonnet, Opus and GPT pipette-mix competent cells after adding DNA on ecoli-hard, which harms +**Error analysis.** Four kinds of error account for every lost point that the audited verifier upholds. *Inventing steps the paper does not describe* +(every model; Qwen fails `fidelity_to_paper` on all four paper-only tasks): Sonnet, Opus and GPT pipette-mix competent cells after adding DNA on ecoli-hard, which harms fragile cells; GPT adds a manual transfer of all 96 PCR reactions to a second plate and a 1 µL water "QC aliquot" in -Golden Gate. *Overdrawing a reservoir*: Sonnet maps six sample columns onto -four ethanol wells, drawing 16 mL from a 15 mL well. *Wrong metadata*: Fable credits the RNA paper to the wrong +Golden Gate. *Overdrawing a reservoir* (Sonnet, Qwen, DeepSeek): six sample columns mapped +onto four ethanol wells draw 16 mL from a 15 mL well. *One tip from the reaction back to the stock* (Qwen on both Golden +Gate tasks, DeepSeek on AMPure and Golden-Gate-hard): `transfer(..., new_tip='once', mix_after=...)` mixes in each +reaction and returns the same tip to the enzyme or bead stock, carrying every reaction into it. *Wrong metadata*: Fable credits the RNA paper to the wrong authors. Two choices that looked like errors are not: recovering the whole 100 µL eluate on RNA-hard and building the -paper's 10 µL colony-PCR reaction both follow the paper; the scores that held them to the authors' 80 µL and the easy -task's 20 µL are provisional. +paper's 10 µL colony-PCR reaction both follow the paper, and every trial graded before the audit has been re-judged. + +**One attempt is noisy, and the cloud is faster.** Qwen run a second time on Modal, everything else unchanged, scored +0.805 against 0.728 locally; four tasks changed, by up to 0.70 (RNA extraction 0.30 to 1.00). Gaps of under about 0.1 +between models are within this noise. With all 11 tasks in parallel Modal finished in 31.9 minutes against 54.7 for local +Docker two at a time, despite about 1.5 minutes of image build per sandbox; the wall clock is then the slowest trial, and +DeepSeek, without a slow trial, finished all 11 in 12 minutes. ## 8 Discussion and limitations **The paper is the hard part.** Easy tasks saturate; the paper-only tasks separate models, and their dominant failure -is inventing steps or quantities. GPT-6.1 Sol, perfect on easy tasks, failed `fidelity_to_paper` on all four -paper-only ones (two provisional). The paper-only level is the one to grow, and each new task must pass the audit. +is inventing steps or quantities. GPT-6.1 Sol, perfect on easy tasks, failed `fidelity_to_paper` on three of the +four paper-only ones, and Qwen on all four. The paper-only level is the one to grow, and each new task must pass the audit. -**Verifiers need adversaries.** Eight grader faults survived reference solutions, unit tests and an adversarial suite, -and surfaced only when we read what real models wrote or audited the ground truth against the paper. One was a false negative that capped a correct protocol at 0.30. +**Verifiers need adversaries.** Ten grader faults survived reference solutions, unit tests and an adversarial suite, +and surfaced only when we read what real models wrote or audited the ground truth against the paper. Three were false negatives that capped or zeroed a correct protocol, two of them found only when new models wrote code in styles the earlier models had not. Benchmarks that grade generated laboratory code should publish their graded artefacts so faults like these can be found. -**Judge noise is real.** Three calls per protocol disagreed on seven items across the run, and single calls had scored +**Judge noise is real.** Three calls per protocol disagreed on 9 of 380 items across the run, and single calls had scored identical recipes differently. Majority voting absorbs this, but all signal still comes from one judge, a Claude model that also judges a non-Claude model. A second judge from another provider, as AEGIS does with five backends [@setty2026aegis], would show whether that matters. **Ground truth comes from the easy task.** Three of the four paper-only tasks reuse their easy task's ground truth, -which is fair only where the paper fixes the same quantities. The audit now enforces that. - -**Provisional scores.** Five scores were judged before the audit or with fewer than three votes: Sonnet and GPT on -RNA-hard and colony-PCR-hard, and Sonnet on heat-shock-hard. They are marked † in Table 4 and on the leaderboard, with -reasons in the released results. API credit ran out before they could be re-judged; an open-weight model (Qwen3.8) is -the next run. +which is fair only where the paper fixes the same quantities. The audit now enforces that; where the paper fixes less, +as in colony-PCR-hard, the checks fall back to what it does fix. -**Scope.** One robot (OT-2), one simulator, one agent harness, four models, one attempt per cell. Repeated attempts, +**Scope.** One robot (OT-2), one simulator, one agent harness, six models, one attempt per cell (one model repeated). Repeated attempts for every model, wet-lab execution of the generated protocols and non-Opentrons instruments are future work. ## 9 Conclusion @@ -356,7 +368,7 @@ Text2WetLab measures whether agents turn lab protocols and papers into robot cod not merely code that runs. Frontier models clear the mechanical layers and nearly solve step-by-step tasks, but when they must read the paper they add steps and quantities of their own. Safety refusals on routine molecular biology are common enough that a benchmark has to keep them out of its scores and report them on their own. And the grader is -itself an artefact to validate: ours needed eight fixes, found only by reading real model output and auditing the +itself an artefact to validate: ours needed ten fixes, found only by reading real model output and auditing the ground truth against the papers. ## Data and code availability diff --git a/results/adversarial.json b/results/adversarial.json index f3eedca..da54c31 100644 --- a/results/adversarial.json +++ b/results/adversarial.json @@ -558,25 +558,24 @@ "error": "" }, "drop_last_transfer": { - "reward": 0.3333, + "reward": 0.25, "failed_checks": [ - "end_state:pcr_plate" + "composition:pcr_plate" ], "lint": [], "error": "" }, "halve_volumes": { - "reward": 0.3333, - "failed_checks": [ - "end_state:pcr_plate" - ], + "reward": 1.0, + "failed_checks": [], "lint": [], - "error": "" + "error": "", + "valid": "9 uL mix + 0.5 + 0.5 = 10 uL, the paper's OT-2 reaction volume" }, "double_volumes": { - "reward": 0.3333, + "reward": 0.25, "failed_checks": [ - "end_state:pcr_plate" + "composition:pcr_plate" ], "lint": [], "error": "" @@ -604,7 +603,7 @@ "error": "" }, "reservoir_labelled_as_plate_kind": { - "reward": 0.1875, + "reward": 0.1667, "failed_checks": [ "deck_labware:master_mix_reservoir", "no_aspirate_from_empty_well" @@ -613,7 +612,7 @@ "error": "" }, "wrong_slot": { - "reward": 0.1667, + "reward": 0.125, "failed_checks": [ "deck_labware:colony_plate", "no_aspirate_from_empty_well" @@ -636,9 +635,9 @@ "error": "" }, "extra_liquid_into_target": { - "reward": 0.3333, + "reward": 0.25, "failed_checks": [ - "end_state:pcr_plate" + "composition:pcr_plate" ], "lint": [], "error": "" diff --git a/results/repeats/2026-10-07-qwen-modal/README.md b/results/repeats/2026-10-07-qwen-modal/README.md new file mode 100644 index 0000000..9fbb8d1 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/README.md @@ -0,0 +1,39 @@ +# Qwen3.8-2.4T-A95B, repeated on Modal (2026-10-07) + +The same model, agent (Claude Code via OpenRouter), tasks and grader as the Qwen run in +[`results/runs/2026-10-07-openrouter`](../../runs/2026-10-07-openrouter/), run a second time on Modal with all 11 +tasks at once (`harbor run ... -e modal -n 11`) instead of local Docker two at a time. It measures two things: what +running in parallel on Modal buys, and how much one model's score moves between two pass@1 attempts. The leaderboard +uses the local run, so every model there ran in the same environment except DeepSeek V4 Pro (Modal only). + +| Job | Trials | Wall clock | Median trial | Median env setup | Median agent | Median verifier | Agent cost | Mean reward (answered) | +|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| Qwen, local Docker (2 at a time) | 11 | 54.7 min | 4.4 min | 7 s | 3.0 min | 46 s | $2.83 | 0.728 (n=11) | +| Qwen, Modal (11 at a time) | 11 | 31.9 min | 7.8 min | 1.5 min | 4.9 min | 45 s | $3.61 | 0.805 (n=11) | +| DeepSeek, Modal (11 at a time) | 11 | 12.0 min | 4.7 min | 1.5 min | 1.8 min | 53 s | $2.53 | 0.581 (n=11) | + +| Task | Qwen, local Docker (2 at a time) | Qwen, Modal (11 at a time) | DeepSeek, Modal (11 at a time) | +|---|:---:|:---:|:---:| +| `a1-a12-100ul` | 1.00 | 1.00 | 0.00 | +| `ampure-bead-cleanup` | 1.00 | 1.00 | 0.30 | +| `colony-pcr-screening` | 1.00 | 1.00 | 1.00 | +| `ecoli-heat-shock-transformation` | 1.00 | 1.00 | 1.00 | +| `golden-gate-assembly` | 0.30 | 0.30 | 1.00 | +| `opentrons-rna-extraction` | 0.30 | 1.00 | 0.30 | +| `split-200ul-two-wells` | 1.00 | 1.00 | 1.00 | +| `colony-pcr-screening-hard` | 0.67 | 0.75 | 0.30 | +| `ecoli-heat-shock-transformation-hard` | 0.75 | 0.30 | 0.75 | +| `golden-gate-assembly-hard` | 0.30 | 0.75 | 0.30 | +| `opentrons-rna-extraction-hard` | 0.69 | 0.75 | 0.44 | + +Made with `scripts/compare_jobs.py ... --prices`. Rewards are as graded at run time: DeepSeek's two grader false +positives (`a1-a12-100ul` 0.00 and `colony-pcr-screening-hard` 0.30 here) were fixed and regraded to 1.00 in the main run. + +- **Wall clock.** Modal finished in 31.9 min against 54.7 min locally (1.7x). With every task running at once the wall + clock is the slowest trial: Qwen's 28-minute agent session on `opentrons-rna-extraction`. DeepSeek, with no such + outlier, finished all 11 in 12.0 min. +- **Per-trial overhead.** Modal builds each task's image in its sandbox (median 1.5 min against 7 s locally, where + images are cached), and its agent sessions ran longer (median 4.9 against 3.0 min). +- **Run-to-run variation.** The mean moved 0.728 to 0.805 between two attempts of the same model. Four tasks changed + score: RNA extraction 0.30 to 1.00, heat-shock-hard 0.75 to 0.30, Golden-Gate-hard 0.30 to 0.75, colony-PCR-hard + 0.67 to 0.75. Differences of this size between models on single attempts are within noise. diff --git a/results/repeats/2026-10-07-qwen-modal/prices.json b/results/repeats/2026-10-07-qwen-modal/prices.json new file mode 100644 index 0000000..d5f436b --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/prices.json @@ -0,0 +1,7 @@ +{ + "qwen/qwen3.8-2.4t-a95b": { + "completion": "0.000006", + "input_cache_read": "0.00000025", + "prompt": "0.000002" + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/grader.json new file mode 100644 index 0000000..a7aa0b0 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/grader.json @@ -0,0 +1,137 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 28 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 is valid; distribute(100,...) with the p300 keeps every aspirate at or below 300 \u00b5L, and the checks for overdispense and empty-well aspiration passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "distribute defaults to one tip for the run, which is dropped at the end; tip_before_aspirate, tip_dropped_at_end and no_cross_contamination all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "The only step, 100 \u00b5L from reservoir A1 to plate A1\u2013A12, is done in order; the extra disposal volume is blown back out and does not change any well's contents (end_state:plate passed).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "deck_and_hardware", + "score": 1, + "evidence": "Reservoir is loaded in slot 1 and plate in slot 2 with the specified labels, p20 is on the left and p300 on the right, and the tip racks are in slots 10 and 11.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "volumes_and_wells", + "score": 1, + "evidence": "pipette=p300 and plate.rows()[0] covers A1\u2013A12 with 100 \u00b5L each from reservoir A1; end_state:plate passed.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "deck_and_hardware": 1.0, + "volumes_and_wells": 1.0 + }, + "summary": "The protocol uses p300 distribute() from reservoir A1 to plate row A (A1\u2013A12), 100 \u00b5L each. It simulates cleanly, all measured checks passed, and the labware, pipettes and slots match the task.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "deck_and_hardware": 0.125, + "volumes_and_wells": 0.125 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/protocol.py new file mode 100644 index 0000000..1b0d9e8 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/protocol.py @@ -0,0 +1,32 @@ +from opentrons import protocol_api + +metadata = { + 'protocolName': 'Distribute 100 uL from reservoir to plate A1:A12', + 'apiLevel': '2.15', +} + + +def run(protocol: protocol_api.ProtocolContext): + # Load labware + reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 1, label='reservoir') + plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='plate') + + # Load pipettes + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', + tip_racks=[protocol.load_labware( + 'opentrons_96_tiprack_20ul', 10)]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', + tip_racks=[protocol.load_labware( + 'opentrons_96_tiprack_300ul', 11)]) + + # 100 uL needs the 300 uL pipette + pipette = p300 + + # Step 1: Transfer 100 uL from reservoir A1 to plate wells A1:A12. + # One source well feeds every destination well; distribute() chunks + # the aspirates so the tip never holds more than the pipette can. + pipette.distribute(100, reservoir['A1'], plate.rows()[0]) diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/reward.json new file mode 100644 index 0000000..5b16841 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/trial.json new file mode 100644 index 0000000..5f7cd9c --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/a1-a12-100ul/trial.json @@ -0,0 +1,29 @@ +{ + "task": "a1-a12-100ul", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + }, + "exception": null, + "cost_usd": 0.05867, + "input_tokens": 68097, + "output_tokens": 1602, + "started_at": "2026-10-07T18:21:21.477734Z", + "finished_at": "2026-10-07T18:25:18.262972Z", + "cache_tokens": 49792, + "claude_code_cost_usd": 0.17932225 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/grader.json new file mode 100644 index 0000000..b35593a --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/grader.json @@ -0,0 +1,168 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 7118 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:sample_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:elution_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, labware loaded with the required labels, and the p300 handles every volume (40\u2013200 \u00b5L). Mix volumes (80 and 45 \u00b5L) fit within the well contents, and all end_state checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Every add_to_plate, remove_to_waste and transfer_pairs call does pick_up_tip and drop_tip for each well. The tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All 13 steps run in order with the right volumes, 10x mixes after the beads and the water, and protocol.comment records for the incubation, magnet, drying and disengage steps. Nothing extra is added and the comments match the actions.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding", + "score": 1, + "evidence": "add_to_plate(40, bead_source, sample_wells, mix_reps=10, mix_volume=80) adds beads and mixes 10x. A 5 min incubation comment follows before the magnet comment.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "supernatant_and_washes", + "score": 1, + "evidence": "A comment records engaging the magnet and waiting 5 min. remove_to_waste(90) clears every well, then two rounds of add_to_plate(200) ethanol and remove_to_waste(200) follow.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "drying_and_elution", + "score": 1, + "evidence": "Comments record a 5 min air dry and disengaging the magnet. 50 \u00b5L water is added with a 10x mix, then comes the 2 min incubation and magnet re-engage comment.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "recovery", + "score": 1, + "evidence": "transfer_pairs(45, sample_wells, elution_wells) pairs the wells in order, A1 to A1 and so on, using a fresh tip each time.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "binding": 1.0, + "supernatant_and_washes": 1.0, + "drying_and_elution": 1.0, + "recovery": 1.0 + }, + "summary": "The protocol follows all 13 steps in order with correct volumes, a fresh tip per well, and comment records for the manual steps. All measured checks passed and I found no deviations.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "binding": 0.0625, + "supernatant_and_washes": 0.0625, + "drying_and_elution": 0.0625, + "recovery": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_binding": 1.0, + "rubric_supernatant_and_washes": 1.0, + "rubric_drying_and_elution": 1.0, + "rubric_recovery": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/protocol.py new file mode 100644 index 0000000..1b624b1 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/protocol.py @@ -0,0 +1,138 @@ +from opentrons import protocol_api + +metadata = { + 'apiLevel': '2.13', + 'protocolName': 'AMPure XP PCR Cleanup', + 'description': '0.8x AMPure XP magnetic bead cleanup of 96 PCR products', +} + + +def run(protocol: protocol_api.ProtocolContext): + # ---------------- Labware ---------------- + sample_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='sample_plate') + beads_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 2, label='beads_reservoir') + ethanol_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='ethanol_reservoir') + water_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 4, label='water_reservoir') + waste = protocol.load_labware( + 'nest_1_reservoir_195ml', 5, label='waste') + elution_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='elution_plate') + + # ---------------- Pipettes ---------------- + tips_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips_300]) + + # All transfers in this protocol are 40-200 uL -> p300 is used throughout. + + # ---------------- Locations ---------------- + sample_wells = sample_plate.wells() # A1..H12, in order + elution_wells = elution_plate.wells() # A1..H12, in order + bead_source = beads_reservoir['A1'] + ethanol_source = ethanol_reservoir['A1'] + water_source = water_reservoir['A1'] + waste_dest = waste['A1'] + + # ---------------- Helpers ---------------- + def add_to_plate(volume, source, dests, mix_reps=0, mix_volume=0): + """One source -> many destination wells, one fresh tip per well.""" + for dest in dests: + p300.pick_up_tip() + p300.aspirate(volume, source) + p300.dispense(volume, dest) + if mix_reps: + p300.mix(mix_reps, mix_volume, dest) + p300.drop_tip() + p300.reset_tipracks() + + def remove_to_waste(volume, sources, dest): + """Many source wells -> one destination, one fresh tip per well.""" + for src in sources: + p300.pick_up_tip() + p300.aspirate(volume, src) + p300.dispense(volume, dest) + p300.drop_tip() + p300.reset_tipracks() + + def transfer_pairs(volume, sources, dests): + """Paired well-to-well transfers (A1->A1, A2->A2, ...).""" + for src, dest in zip(sources, dests): + p300.pick_up_tip() + p300.aspirate(volume, src) + p300.dispense(volume, dest) + p300.drop_tip() + p300.reset_tipracks() + + # ---------------- Protocol ---------------- + + # Step 1: Add 40 uL AMPure XP beads to each sample well, mix 10x. + # Wells hold 50 uL PCR product + 40 uL beads = 90 uL; mix below that. + protocol.comment('Step 1: Adding 40 uL AMPure XP beads to each sample ' + 'well and mixing 10 times.') + add_to_plate(40, bead_source, sample_wells, mix_reps=10, mix_volume=80) + + # Step 2: Incubation (manual step, not simulated). + protocol.comment('Step 2: Incubate sample plate 5 minutes at room ' + 'temperature (beads bind DNA).') + + # Step 3: Magnet (manual step, not simulated). + protocol.comment('Step 3: Engage magnetic module; wait 5 minutes until ' + 'solution clears.') + + # Step 4: Remove 90 uL supernatant from each well to waste. + protocol.comment('Step 4: Removing 90 uL supernatant from each well ' + 'to waste.') + remove_to_waste(90, sample_wells, waste_dest) + + # Step 5: Wash 1 - add 200 uL 80% ethanol to each well. + protocol.comment('Step 5: Wash 1 - adding 200 uL 80% ethanol to each ' + 'well.') + add_to_plate(200, ethanol_source, sample_wells) + + # Step 6: Remove 200 uL ethanol wash 1 to waste. + protocol.comment('Step 6: Removing 200 uL ethanol wash 1 to waste.') + remove_to_waste(200, sample_wells, waste_dest) + + # Step 7: Wash 2 - add 200 uL 80% ethanol to each well. + protocol.comment('Step 7: Wash 2 - adding 200 uL 80% ethanol to each ' + 'well.') + add_to_plate(200, ethanol_source, sample_wells) + + # Step 8: Remove 200 uL ethanol wash 2 to waste. + protocol.comment('Step 8: Removing 200 uL ethanol wash 2 to waste.') + remove_to_waste(200, sample_wells, waste_dest) + + # Step 9: Air dry (manual step, not simulated). + protocol.comment('Step 9: Air dry beads 5 minutes at room temperature ' + '(magnet engaged); beads should appear matte, not ' + 'shiny.') + + # Step 10: Disengage magnet (manual step, not simulated). + protocol.comment('Step 10: Disengage magnetic module.') + + # Step 11: Elute - add 50 uL nuclease-free water, mix 10x. + protocol.comment('Step 11: Adding 50 uL nuclease-free water to each ' + 'well and mixing 10 times.') + add_to_plate(50, water_source, sample_wells, mix_reps=10, mix_volume=45) + + # Step 12: Incubation + magnet (manual steps, not simulated). + protocol.comment('Step 12: Incubate sample plate 2 minutes at room ' + 'temperature; then re-engage magnetic module for 5 ' + 'minutes.') + + # Step 13: Transfer 45 uL cleaned DNA to elution plate (paired wells). + protocol.comment('Step 13: Transferring 45 uL cleaned DNA from sample ' + 'plate to elution plate.') + transfer_pairs(45, sample_wells, elution_wells) + + protocol.comment('AMPure XP cleanup complete. Elution plate holds 45 uL ' + 'cleaned DNA per well; 5 uL remains in each sample ' + 'well with the beads.') diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/reward.json new file mode 100644 index 0000000..48d40a0 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_binding": 1.0, "rubric_supernatant_and_washes": 1.0, "rubric_drying_and_elution": 1.0, "rubric_recovery": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/trial.json new file mode 100644 index 0000000..54e1c26 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ampure-bead-cleanup/trial.json @@ -0,0 +1,31 @@ +{ + "task": "ampure-bead-cleanup", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_binding": 1.0, + "rubric_supernatant_and_washes": 1.0, + "rubric_drying_and_elution": 1.0, + "rubric_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.114321, + "input_tokens": 111955, + "output_tokens": 7179, + "started_at": "2026-10-07T18:21:21.526205Z", + "finished_at": "2026-10-07T18:28:21.237305Z", + "cache_tokens": 87236, + "claude_code_cost_usd": 0.37754925 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/grader.json new file mode 100644 index 0000000..b19ec0f --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/grader.json @@ -0,0 +1,157 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1997 + }, + "not_applicable": [ + "end_state:pcr_plate (the source paper does not determine these volumes; judged by the rubric)" + ], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:colony_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13 with metadata; all volumes (5, 4, 1 \u00b5L) go through the p20 within 1\u201320 \u00b5L, and the simulator ran with no overdispense or empty-well checks failing.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Master mix is distributed with one tip (new_tip='once'), and primers and template use new_tip='always'. The tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks all pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "The cycling program uses a 5 min initial denaturation and 35 cycles, which departs from the Q5 manufacturer's 30 s start and about 30 cycles. The code also adds an invented manual spin-down step and assumes the primer plate is pre-diluted.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "reaction_setup", + "score": 1, + "evidence": "Each well gets 5 \u00b5L 2x Q5 master mix (1x final), 4 \u00b5L of its primer pair and 1 \u00b5L colony template, added in that order. That makes a 10 \u00b5L reaction, matching the paper's OT-2 volume.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "sample_mapping", + "score": 1, + "evidence": "colony_plate.wells()[:96], primer_plate.wells()[:96] and pcr_plate.wells()[:96] are paired by index in the transfers, so A1 maps to A1 and so on, with no well skipped or reused.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "thermocycling", + "score": 1, + "evidence": "A comment records sealing and a program of 98 \u00b0C initial denaturation, 35 cycles of 98/60/72 \u00b0C, 72 \u00b0C final extension and a 4 \u00b0C hold, which is suitable for Q5 Hot Start.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "reaction_setup": 1.0, + "sample_mapping": 1.0, + "thermocycling": 1.0 + }, + "summary": "The protocol sets up the 96 reactions correctly: 5 \u00b5L 2x master mix, 4 \u00b5L primer pair and 1 \u00b5L template per well, with correct well mapping and tip handling, and the simulator checks pass. It fails the all-or-nothing fidelity item because of deviations in the cycling program (5 min initial denaturation, 35 cycles) and an invented spin-down step.", + "level": "hard", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "reaction_setup": 0.0833, + "sample_mapping": 0.0833, + "thermocycling": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/protocol.py new file mode 100644 index 0000000..1a2a539 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/protocol.py @@ -0,0 +1,166 @@ +"""Colony PCR screening of 96 transformant colonies on the Opentrons OT-2. + +Adapted from the OT-2 colony PCR workflow of the Slowpoke paper: + Slowpoke: An Automated Golden Gate Cloning Workflow for Opentrons OT-2 + and Flex, ACS Synth. Biol., doi:10.1021/acssynbio.5c00629 + (Materials & Methods 2.5, "Automated Colony PCR"). + +Paper's OT-2 colony PCR (Section 2.5): + A master mix (water + colony PCR primers + 2x PCR master mix) is + dispensed at 9 uL per reaction, then 1 uL of colony template is added, + giving a 10 uL reaction. Cycling follows the polymerase manufacturer's + protocol, with annealing temperature and extension time adjusted for + the primers and amplicon. + +Adaptations to the fixed deck of this run (each choice noted below): + * The 2x master mix (Q5 Hot Start) is already pre-mixed in a 1-well + reservoir; the robot therefore cannot prepare the paper's master mix + from separate water/primer/mix stocks and dispenses the 2x mix directly. + * Each colony has its OWN primer pair in primer_plate (same well position + as the colony), so primers cannot be part of a common master mix and are + added individually to each reaction. + * The reaction stays at the paper's OT-2 volumes: 10 uL total with 1 uL + of colony template. The paper's 9 uL of master mix is split into + 5 uL of 2x master mix (gives 1x final) + 4 uL of primer pair solution. + This assumes the primer plate holds each primer pair pre-diluted at a + working concentration such that 4 uL in 10 uL gives the desired final + primer concentration (the paper leaves primer concentrations open). + * No thermocycler module is loaded on this deck, so thermocycling is a + manual benchtop step, recorded with protocol.comment() at the end (the + paper ran the OT-2 workflow on the thermocycler module with benchtop + cyclers added as needed). +""" + +metadata = { + 'protocolName': 'Colony PCR screen of 96 colonies (Slowpoke OT-2 workflow, Q5 master mix)', + 'author': 'Generated from doi:10.1021/acssynbio.5c00629', + 'description': 'Set up 96 colony PCR reactions (Q5 Hot Start 2x master mix, ' + 'per-colony primer pairs, 1 uL colony template) following the ' + 'Slowpoke OT-2 colony PCR workflow.', + 'apiLevel': '2.13', +} + +# --- Volumes (per 10 uL reaction, paper's OT-2 values) --------------------- +TEMPLATE_VOLUME = 1 # uL colony template (paper, OT-2 platform) +MASTER_MIX_VOLUME = 5 # uL Q5 Hot Start 2x master mix (1x final) +PRIMER_VOLUME = 4 # uL primer pair solution; with the master mix this + # replaces the paper's 9 uL premixed master mix +TOTAL_VOLUME = MASTER_MIX_VOLUME + PRIMER_VOLUME + TEMPLATE_VOLUME # = 10 uL + +N_COLONIES = 96 + + +def run(protocol): + # --- Labware (fixed deck layout) --------------------------------------- + colony_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='colony_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='pcr_plate') + master_mix_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='master_mix_reservoir') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='primer_plate') + + tiprack_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tiprack_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tiprack_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tiprack_300]) + + # All liquid volumes in this protocol are 1-20 uL, so only the p20 is + # used. The p300 is loaded because the deck layout is fixed, but it + # stays idle. + del p300 + + master_mix_source = master_mix_reservoir.wells_by_name()['A1'] + + # One source well per colony, mapped 1:1 by well position: + # colony_plate[i] -> primer_plate[i] -> pcr_plate[i] + colony_wells = colony_plate.wells()[:N_COLONIES] + primer_wells = primer_plate.wells()[:N_COLONIES] + pcr_wells = pcr_plate.wells()[:N_COLONIES] + + protocol.comment( + f"Set up {N_COLONIES} colony PCRs of {TOTAL_VOLUME} uL each: " + f"{MASTER_MIX_VOLUME} uL Q5 Hot Start 2x master mix + " + f"{PRIMER_VOLUME} uL primer pair (per colony) + " + f"{TEMPLATE_VOLUME} uL colony template. " + "Well map: colony_plate well X -> primer_plate well X -> pcr_plate well X.") + + # MANUAL STEP: the robot cannot unseal plates or check reagents. + protocol.comment( + "MANUAL STEP before start: remove lids/seals from colony_plate, " + "primer_plate and pcr_plate; confirm master_mix_reservoir contains " + "thawed, mixed Q5 Hot Start 2x master mix (>= ~500 uL incl. dead " + "volume). Q5 Hot Start tolerates room-temperature set-up.") + + # --- Step 1: distribute 2x master mix ----------------------------------- + # Same reagent for every reaction, so one tip is re-used for the whole + # distribution (as in the paper's master-mix distribution step). + protocol.comment( + f"Distributing {MASTER_MIX_VOLUME} uL of Q5 Hot Start 2x master mix " + f"from master_mix_reservoir A1 to all {N_COLONIES} wells of pcr_plate.") + p20.distribute( + MASTER_MIX_VOLUME, + master_mix_source, + pcr_wells, + new_tip='once', + ) + + # One tip per well is used from here on; start each step from a full rack. + p20.reset_tipracks() + + # --- Step 2: add each colony's primer pair ------------------------------ + protocol.comment( + f"Adding {PRIMER_VOLUME} uL of each primer pair from primer_plate to " + "the same well of pcr_plate (fresh tip per well).") + p20.transfer( + PRIMER_VOLUME, + primer_wells, + pcr_wells, + new_tip='always', # different primer pair in every well + touch_tip=True, + ) + + p20.reset_tipracks() + + # --- Step 3: add colony template (paper: template added last) ----------- + # MANUAL STEP: the robot cannot spin plates. + protocol.comment( + "MANUAL STEP: briefly spin down colony_plate so all template is at " + "the bottom of the wells and colonies are resuspended.") + protocol.comment( + f"Adding {TEMPLATE_VOLUME} uL of colony template from colony_plate to " + "the same well of pcr_plate (fresh tip per well), then mixing.") + p20.transfer( + TEMPLATE_VOLUME, + colony_wells, + pcr_wells, + new_tip='always', + touch_tip=True, + mix_after=(3, 8), # gentle mix of the 10 uL reaction after addition + ) + + # --- Manual thermocycling ------------------------------------------------ + # MANUAL STEP: no thermocycler module is loaded on this deck, and the OT-2 + # cannot seal plates or move them to a cycler. The program below follows + # the NEB Q5 Hot Start 2x master mix protocol with colony-PCR adjustments; + # the paper likewise leaves cycling to the polymerase manufacturer's + # protocol with primer/amplicon-specific annealing and extension. + protocol.comment( + "MANUAL STEP - thermocycling: seal pcr_plate with an adhesive PCR " + "seal, spin down, and run it in a benchtop thermocycler " + "(heated lid ~105 degC):\n" + " 98 degC 5 min initial denaturation / colony lysis / hot-start " + "activation (NEB gives 30 s; extended here because template is crude " + "colony material)\n" + " 35 cycles of:\n" + " 98 degC 10 s denaturation\n" + " 60 degC 30 s annealing (adjust to the primer Tms)\n" + " 72 degC 30 s extension (scale to 20-30 s/kb of amplicon)\n" + " 72 degC 2 min final extension\n" + " hold at 4 degC\n" + "Then analyse the PCR products by agarose gel electrophoresis to " + "score the colonies.") diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/reward.json new file mode 100644 index 0000000..c32e10c --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.75, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/trial.json new file mode 100644 index 0000000..b7c57e8 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/trial.json @@ -0,0 +1,30 @@ +{ + "task": "colony-pcr-screening-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 1.0 + }, + "exception": null, + "cost_usd": 0.310951, + "input_tokens": 376353, + "output_tokens": 17735, + "started_at": "2026-10-07T18:21:21.470693Z", + "finished_at": "2026-10-07T18:32:52.867783Z", + "cache_tokens": 313237, + "claude_code_cost_usd": 0.9559497499999999 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/grader.json new file mode 100644 index 0000000..3ac1b10 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/grader.json @@ -0,0 +1,160 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1831 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:colony_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, all volumes (18, 1, 10 uL mix) are within p20 range, labware loaded with the required labels, and the measured checks passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Every transfer uses new_tip='always'; tip_before_aspirate, no_cross_contamination and tip_dropped_at_end all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All four steps run in order with the specified volumes, the colony step mixes 3x after dispensing, and the seal and thermocycling comment is accurate. The only choice not given in the task is the 10 uL mix volume, which is reasonable.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "master_mix", + "score": 1, + "evidence": "p20.transfer(18, master_mix_reservoir['A1'], pcr_wells, new_tip='always') fills all 96 wells, and the pcr_plate end_state check passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "template_and_primers", + "score": 1, + "evidence": "Two p20 transfers of 1 uL pair colony_wells and primer_wells with pcr_wells in order; end-state checks passed for all plates.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "thermocycling", + "score": 1, + "evidence": "protocol.comment records sealing and the full program: 98\u00b0C 30 s; 30x [98\u00b0C 10 s, 60\u00b0C 30 s, 72\u00b0C 30 s]; 72\u00b0C 2 min; 4\u00b0C hold.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "master_mix": 1.0, + "template_and_primers": 1.0, + "thermocycling": 1.0 + }, + "summary": "The protocol simulates cleanly and all measured checks pass. It follows the task's steps, volumes and order, uses fresh tips throughout, and records the seal and thermocycling program, so all items score 1.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "master_mix": 0.0833, + "template_and_primers": 0.0833, + "thermocycling": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_master_mix": 1.0, + "rubric_template_and_primers": 1.0, + "rubric_thermocycling": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/protocol.py new file mode 100644 index 0000000..acbc53b --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/protocol.py @@ -0,0 +1,101 @@ +"""Colony PCR screening with Q5 Hot Start master mix (OT-2). + +Screen transformant colonies by PCR: dispense 18 uL of Q5 Hot Start master +mix into each well of a 96-well PCR plate, add 1 uL of colony template per +well (mix after dispensing), then add 1 uL of the corresponding primer pair. +Based on the Slowpoke colony PCR workflow (ACS Synth. Biol., +doi:10.1021/acssynbio.5c00629). +""" + +from opentrons import protocol_api + +metadata = { + 'apiLevel': '2.13', + 'protocolName': 'Colony PCR screening with Q5 Hot Start master mix', + 'description': '96-well colony PCR setup: 18 uL Q5 master mix, ' + '1 uL colony template (mix 3x), 1 uL primer pair per well.', +} + + +def run(protocol: protocol_api.ProtocolContext): + # ------------------------------------------------------------------ + # Labware (fixed deck layout) + # ------------------------------------------------------------------ + colony_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='colony_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='pcr_plate') + master_mix_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='master_mix_reservoir') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='primer_plate') + + # Tip racks + tips_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + # Pipettes: p20 for 1-20 uL, p300 for 20-300 uL + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips_300]) + + # Wells A1:H12 in plate order (A1, A2, ..., A12, B1, ..., H12); + # source and destination wells pair in this order. + colony_wells = colony_plate.wells() + primer_wells = primer_plate.wells() + pcr_wells = pcr_plate.wells() + + # ------------------------------------------------------------------ + # Step 1: dispense 18 uL of Q5 Hot Start master mix 2x from the + # reservoir into every well of the PCR plate. + # (18 uL is within the p20 range of 1-20 uL.) + # ------------------------------------------------------------------ + protocol.comment( + 'Step 1: Dispense 18 uL of Q5 Hot Start master mix 2x from ' + 'master_mix_reservoir A1 into all 96 wells of pcr_plate.') + p20.transfer( + 18, + master_mix_reservoir['A1'], + pcr_wells, + new_tip='always', + ) + + # ------------------------------------------------------------------ + # Step 2: add 1 uL of colony template to the corresponding well and + # mix 3 times after dispensing. Fresh tip per well to avoid + # cross-contamination between colonies. + # ------------------------------------------------------------------ + protocol.comment( + 'Step 2: Add 1 uL of colony template from colony_plate to the ' + 'corresponding well of pcr_plate and mix 3 times.') + p20.reset_tipracks() + p20.transfer( + 1, + colony_wells, + pcr_wells, + new_tip='always', + mix_after=(3, 10), + ) + + # ------------------------------------------------------------------ + # Step 3: add 1 uL of primer pair to the corresponding well. + # ------------------------------------------------------------------ + protocol.comment( + 'Step 3: Add 1 uL of primer pair from primer_plate to the ' + 'corresponding well of pcr_plate.') + p20.reset_tipracks() + p20.transfer( + 1, + primer_wells, + pcr_wells, + new_tip='always', + ) + + # ------------------------------------------------------------------ + # Step 4: off-robot steps (sealing and thermocycling) - not simulated. + # ------------------------------------------------------------------ + protocol.comment( + 'Step 4: Seal pcr_plate and thermocycle: 98°C for 30 s; ' + '[98°C for 10 s, 60°C for 30 s, 72°C for 30 s] x 30 cycles; ' + '72°C for 2 min; hold at 4°C.') diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/reward.json new file mode 100644 index 0000000..12f62ba --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_master_mix": 1.0, "rubric_template_and_primers": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/trial.json new file mode 100644 index 0000000..0df33fb --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/colony-pcr-screening/trial.json @@ -0,0 +1,30 @@ +{ + "task": "colony-pcr-screening", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_master_mix": 1.0, + "rubric_template_and_primers": 1.0, + "rubric_thermocycling": 1.0 + }, + "exception": null, + "cost_usd": 0.181538, + "input_tokens": 209252, + "output_tokens": 10647, + "started_at": "2026-10-07T18:21:21.532712Z", + "finished_at": "2026-10-07T18:29:06.701102Z", + "cache_tokens": 171913, + "claude_code_cost_usd": 0.5854477499999998 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/grader.json new file mode 100644 index 0000000..d2b5931 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/grader.json @@ -0,0 +1,157 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 194 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": false, + "detail": "[(15, 'aspirate_from_empty'), (25, 'aspirate_from_empty'), (35, 'aspirate_from_empty')]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plasmid_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:transformation_plate", + "pass": false, + "detail": "expected {'A1': 61.0, 'B1': 61.0, 'C1': 61.0, 'D1': 61.0, 'E1': 61.0, 'F1': 61.0, 'G1': 61.0, 'H1': 61.0}, got {'A1': 61.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0, 'E1': 10.0, 'F1': 10.0, 'G1': 10.0, 'H1': 10.0, 'A2': 56.0, 'A3': 56.0, 'A4': 56.0, 'A5': 56.0, 'A6': 56.0, 'A7': 56.0, 'A8': 56.0}" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 0, + "evidence": "`cell_wells = transformation_plate.rows()[0]` is row A (A1..A12), not column 1, so DNA and SOC went into A2\u2013A8 and the mixes aspirated from empty wells (no_aspirate_from_empty_well failed).", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "A fresh tip is picked up and dropped for each plasmid and each SOC addition; tip_before_aspirate, tip_dropped_at_end and no_cross_contamination all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "The deck mapping is wrong (end_state shows B1\u2013H1 stuck at 10 \u00b5L and A2\u2013A8 at 56 \u00b5L), and unrequested mixing steps were added.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "dna_addition", + "score": 0, + "evidence": "DNA from plasmid wells A1\u2013H1 went into transformation-plate wells A1\u2013A8 (row A) instead of A1\u2013H1, so B1\u2013H1 never received DNA.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "heat_shock", + "score": 1, + "evidence": "The code runs the closed-lid thermocycler program: 4 \u00b0C for 30 min, then 42 \u00b0C for 30 s, then back to 4 \u00b0C.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "soc_recovery", + "score": 0, + "evidence": "50 \u00b5L SOC was dispensed into cell_wells[i], which is A1\u2013A8, so B1\u2013H1 got no SOC; end_state shows 10 \u00b5L in B1\u2013H1 and 56 \u00b5L in A2\u2013A8.", + "votes": [ + 0, + 0, + 0 + ] + } + ], + "scores": { + "robot_practice": 0.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "dna_addition": 0.0, + "heat_shock": 1.0, + "soc_recovery": 0.0 + }, + "summary": "The protocol indexes the transformation plate with rows()[0], which is row A, instead of column 1. DNA and SOC therefore went into the wrong wells, and the mixes aspirated from empty wells. The thermal program and tip handling are correct, but the DNA addition, SOC recovery, robot practice and fidelity items fail.", + "level": "hard", + "score": 0.3333, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "dna_addition": 0.0833, + "heat_shock": 0.0833, + "soc_recovery": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [ + "no_aspirate_from_empty_well" + ], + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.75, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.125, + "judge_score": 0.3333, + "critical_fail": 1.0, + "rubric_robot_practice": 0.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_dna_addition": 0.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 0.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/protocol.py new file mode 100644 index 0000000..c296dc5 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/protocol.py @@ -0,0 +1,140 @@ +"""APEX Protocol 1: automated heat shock transformation on the OT-2 thermocycler. + +Implements the heat shock transformation module of APEX (Kasprzyk, Herrera & +Stracquadanio, "APEX: Automated Protein EXpression in Escherichia coli", +bioRxiv 2024, doi:10.1101/2024.08.13.607171) for 8 plasmids (pEX01-pEX08). + +Parameters taken from the paper: + - Low Volume (LV) transformation configuration, selected by the authors for + all subsequent automated experiments (Results > Optimisation of + transformation volume). Per the Figure 4 caption, the LV/automated run + uses 1 uL of DNA (1.5e-4 pmol), 10 uL of cells and 50 uL of SOC medium. + - Temperature program (Results > Protocol 1): DNA incubated at 4 C for + 30 minutes, heat shock at 42 C for 30 seconds, recovery with SOC at 37 C + for 1 hour. All steps run on the thermocycler module, which the paper + chose over a temperature module for its rapid temperature changes. + +Choices made where the paper leaves things open (documented here as required): + - The paper does not mention a post-shock ice step; after the 42 C shock the + block is set back to 4 C while SOC is added, keeping cells cold. + - The paper does not specify mixing; a gentle pipette mix after adding DNA + and after adding SOC is performed to homogenise each transformation. + - A heated lid (105 C, the thermocycler default used to prevent + evaporation/condensation during closed-lid incubations) is kept on for + every closed-lid step, matching the paper's rationale for using the + thermocycler instead of the temperature module. + - One fresh tip per plasmid to avoid cross-contamination. + +Deck setup (fixed by the operator): + slot 1: biorad_96_wellplate_200ul_pcr 'plasmid_plate' + A1..H1: plasmids pEX01..pEX08, 10 uL each (1.5e-4 pmol/uL) + slot 2: nest_12_reservoir_15ml 'soc_reservoir', A1: SOC medium + slots 7/8/10/11: Thermocycler Module GEN1 holding + biorad_96_wellplate_200ul_pcr 'transformation_plate' + A1..H1: competent E. coli DH5a, 10 uL each (pre-chilled block) + slot 4: opentrons_96_tiprack_20ul, slot 5: opentrons_96_tiprack_300ul + left: p20_single_gen2 (1-20 uL), right: p300_single_gen2 (20-300 uL) +""" + +metadata = { + 'protocolName': 'APEX Protocol 1: heat shock transformation (LV configuration)', + 'author': 'Generated from Kasprzyk et al., bioRxiv 2024, doi:10.1101/2024.08.13.607171', + 'apiLevel': '2.15', +} + +# LV configuration volumes (paper: 1 uL DNA + 10 uL cells + 50 uL SOC). +DNA_VOLUME = 1 # uL, plasmid DNA (1.5e-4 pmol per transformation) +CELL_VOLUME = 10 # uL, competent cells already in the plate +SOC_VOLUME = 50 # uL, SOC recovery medium +SHOCK_VOLUME = DNA_VOLUME + CELL_VOLUME # 11 uL during incubation/shock +RECOVERY_VOLUME = SHOCK_VOLUME + SOC_VOLUME # 61 uL during recovery + +# Temperature program (paper's Protocol 1 defaults). +INCUBATION_TEMP = 4 # C +INCUBATION_MIN = 30 +SHOCK_TEMP = 42 # C +SHOCK_SEC = 30 +RECOVERY_TEMP = 37 # C +RECOVERY_MIN = 60 + +PLASMID_WELLS = ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1', 'H1'] + + +def run(protocol): + # --- Labware and instruments (fixed deck layout) --------------------- + plasmid_plate = protocol.load_labware( + 'biorad_96_wellplate_200ul_pcr', 1, label='plasmid_plate') + soc_reservoir = protocol.load_labware( + 'nest_12_reservoir_15ml', 2, label='soc_reservoir') + + tc = protocol.load_module('thermocycler') + transformation_plate = tc.load_labware( + 'biorad_96_wellplate_200ul_pcr', label='transformation_plate') + + tips_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 4) + tips_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 5) + + # 1 uL DNA -> p20 (1-20 uL); 50 uL SOC -> p300 (20-300 uL). + p20 = protocol.load_instrument('p20_single_gen2', 'left', + tip_racks=[tips_20]) + p300 = protocol.load_instrument('p300_single_gen2', 'right', + tip_racks=[tips_300]) + + cell_wells = transformation_plate.rows()[0] # A1..H1 + soc = soc_reservoir['A1'] + + # --- Setup: keep the pre-chilled block cold while pipetting ---------- + tc.open_lid() + tc.set_block_temperature(INCUBATION_TEMP) + protocol.comment('Block chilled to 4 C; adding DNA to competent cells.') + + # --- Add DNA: transformation n gets plasmid well n -> cell well n ---- + # Fresh tip per plasmid to avoid cross-contamination; gentle mix after + # dispensing (choice: paper does not specify mixing). + for i, well_name in enumerate(PLASMID_WELLS): + p20.pick_up_tip() + p20.aspirate(DNA_VOLUME, plasmid_plate[well_name]) + p20.dispense(DNA_VOLUME, cell_wells[i]) + p20.mix(3, 5, cell_wells[i]) + p20.drop_tip() + + # --- Closed-lid temperature program: incubation + heat shock --------- + tc.set_lid_temperature(105) # heated lid prevents evaporation/condensation + tc.close_lid() + + protocol.comment('Incubating DNA + cells: 30 min at 4 C.') + tc.set_block_temperature(INCUBATION_TEMP, + hold_time_minutes=INCUBATION_MIN, + block_max_volume=SHOCK_VOLUME) + + protocol.comment('Heat shock: 30 s at 42 C.') + tc.set_block_temperature(SHOCK_TEMP, + hold_time_seconds=SHOCK_SEC, + block_max_volume=SHOCK_VOLUME) + + # Back to 4 C while SOC is added (choice: paper specifies no explicit + # post-shock ice step; this keeps the cells cold during handling). + tc.set_block_temperature(INCUBATION_TEMP, block_max_volume=SHOCK_VOLUME) + tc.open_lid() + + # --- Add SOC recovery medium ------------------------------------------ + for i, well_name in enumerate(PLASMID_WELLS): + p300.pick_up_tip() + p300.aspirate(SOC_VOLUME, soc) + p300.dispense(SOC_VOLUME, cell_wells[i]) + p300.mix(3, 40, cell_wells[i]) # homogenise 61 uL recovery mix + p300.drop_tip() + + # --- Recovery: 1 h at 37 C -------------------------------------------- + tc.close_lid() + protocol.comment('Recovery: 60 min at 37 C with SOC.') + tc.set_block_temperature(RECOVERY_TEMP, + hold_time_minutes=RECOVERY_MIN, + block_max_volume=RECOVERY_VOLUME) + + # --- End state: cells held cold until Protocol 2 (spotting) ---------- + tc.open_lid() + tc.deactivate_lid() + tc.set_block_temperature(INCUBATION_TEMP) + protocol.comment('Transformation complete: recovered cultures held at ' + '4 C. Proceed to APEX Protocol 2 (spotting on agar).') diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/reward.json new file mode 100644 index 0000000..3991ed2 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.75, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.125, "judge_score": 0.3333, "critical_fail": 1.0, "rubric_robot_practice": 0.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_dna_addition": 0.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 0.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/trial.json new file mode 100644 index 0000000..1f61b6a --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/trial.json @@ -0,0 +1,30 @@ +{ + "task": "ecoli-heat-shock-transformation-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.75, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.125, + "judge_score": 0.3333, + "critical_fail": 1.0, + "rubric_robot_practice": 0.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_dna_addition": 0.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 0.0 + }, + "exception": null, + "cost_usd": 0.156076, + "input_tokens": 253659, + "output_tokens": 8802, + "started_at": "2026-10-07T18:21:21.520197Z", + "finished_at": "2026-10-07T18:27:54.602018Z", + "cache_tokens": 230888, + "claude_code_cost_usd": 0.47774525 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/grader.json new file mode 100644 index 0000000..824ca71 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/grader.json @@ -0,0 +1,155 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 10 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plasmid_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:cells_rack", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 valid; 2 \u00b5L via p20 and 250 \u00b5L via p300, correct labware, no overfill, all checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Each aspirate/dispense has a pick_up_tip first and drop_tip after; separate fresh tips for DNA and SOC; tip_dropped_at_end and no_cross_contamination pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All four steps are done in order with correct volumes and conditions; comments accurately record heat shock (42C 45 s, ice 2 min) and incubation (37C 60 min 250 rpm).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dna_addition", + "score": 1, + "evidence": "p20.aspirate(2, plasmid_plate['A1']) then dispense(2, tubes_1_5ml_1['A1']).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "heat_shock", + "score": 1, + "evidence": "protocol.comment records 42C for 45 s then ice 2 min after DNA addition and before SOC.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "soc_recovery", + "score": 1, + "evidence": "p300 transfers 250 \u00b5L SOC from reservoir A1 to tube A1 after heat shock, followed by the comment for 37C, 60 min, 250 rpm.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "dna_addition": 1.0, + "heat_shock": 1.0, + "soc_recovery": 1.0 + }, + "summary": "The protocol simulates cleanly, all measured checks pass, and every step matches the task in order and quantity. Tips are handled correctly, with a separate fresh tip for the DNA and the SOC, and each is dropped.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "dna_addition": 0.0833, + "heat_shock": 0.0833, + "soc_recovery": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/protocol.py new file mode 100644 index 0000000..4ffd0e7 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/protocol.py @@ -0,0 +1,51 @@ +from opentrons import protocol_api + +metadata = { + 'protocolName': 'E. coli heat shock transformation with SOC recovery', + 'apiLevel': '2.15', +} + + +def run(protocol: protocol_api.ProtocolContext): + # Load labware + plasmid_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='plasmid_plate') + tubes_1_5ml_1 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + soc_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='soc_reservoir') + + # Load tip racks + tiprack_20 = protocol.load_labware( + 'opentrons_96_tiprack_20ul', 10) + tiprack_300 = protocol.load_labware( + 'opentrons_96_tiprack_300ul', 11) + + # Load pipettes + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tiprack_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tiprack_300]) + + # Step 1: Transfer 2 uL plasmid DNA from plasmid_plate A1 to tube A1 + p20.pick_up_tip() + p20.aspirate(2, plasmid_plate['A1']) + p20.dispense(2, tubes_1_5ml_1['A1']) + p20.drop_tip() + + # Step 2: Heat shock (not simulated) + protocol.comment( + 'Heat shock cells_rack at 42C for 45 s, then transfer immediately ' + 'to ice for 2 min') + + # Step 3: Transfer 250 uL SOC medium from reservoir A1 to tube A1 + p300.pick_up_tip() + p300.aspirate(250, soc_reservoir['A1']) + p300.dispense(250, tubes_1_5ml_1['A1']) + p300.drop_tip() + + # Step 4: Incubation (not simulated) + protocol.comment( + 'Incubate cells_rack at 37C for 60 min at 250 rpm for outgrowth ' + 'recovery') diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/reward.json new file mode 100644 index 0000000..b650ada --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/trial.json new file mode 100644 index 0000000..648d1b5 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/trial.json @@ -0,0 +1,30 @@ +{ + "task": "ecoli-heat-shock-transformation", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.049608, + "input_tokens": 50569, + "output_tokens": 1086, + "started_at": "2026-10-07T18:21:21.499159Z", + "finished_at": "2026-10-07T18:23:51.065271Z", + "cache_tokens": 33169, + "claude_code_cost_usd": 0.15246200000000001 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/grader.json new file mode 100644 index 0000000..8a17368 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/grader.json @@ -0,0 +1,233 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 557 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:water", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dntp", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_pol", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_mm", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:template_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:rcutsmart", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dpni", + "pass": true, + "detail": "" + }, + { + "name": "end_state:t4_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:gg_enzyme", + "pass": true, + "detail": "" + }, + { + "name": "end_state:assembly_plate", + "pass": false, + "detail": "expected {'A1': 15.0, 'B1': 15.0, 'C1': 15.0, 'D1': 15.0}, got {'A1': 10.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0}" + }, + { + "name": "end_state:cells_plate", + "pass": false, + "detail": "expected {'A1': 305.0, 'B1': 305.0, 'C1': 305.0, 'D1': 305.0}, got {'A1': 310.0, 'B1': 310.0, 'C1': 310.0, 'D1': 310.0}" + }, + { + "name": "end_state:lb_dextrose", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, p20 used for 1-20 uL and p300 for \u226520 uL (106, 40, 250 uL; 30 uL mixes), no overfilled wells, and all simulator checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "tip_before_aspirate, tip_dropped_at_end and no_cross_contamination all pass. Fresh tips are used per primer, template, chromoprotein fragment and eluate, and shared reagents are dispensed with one tip.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "The code transfers the whole 10 uL eluate to the cells (p20.transfer(ASSEMBLY_ELUTION,...)) and skips the NanoDrop 5 uL aliquot. The paper transforms about 5 uL and keeps 5 uL for quantification, and end_state shows assembly_plate 10 vs 15 and cells_plate 310 vs 305.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "pcr_setup", + "score": 1, + "evidence": "Master mix is 13.25 water + 5 buffer + 0.5 dNTP + 0.25 polymerase per reaction (19 uL). Each well then gets 2.5+2.5 uL primers and 1 uL template, with the primer and template mapping from the design table, giving 25 uL.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dpni_and_cleanup", + "score": 1, + "evidence": "The PCR comment records 98 C 30 s, 34 cycles of 98 C 10 s, annealing 30 s and 72 C, then 72 C 5 min. DpnI is 19 uL water + 5 uL rCutSmart + 1 uL DpnI, then 37 C 30 min and 65 C 20 min, followed by a clean-up pause, in that order.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "assembly_mix", + "score": 1, + "evidence": "Each assembly well gets 7 uL water, 2 uL T4 buffer, fragments 3/2/3/2 uL per the design table and 1 uL enzyme mix, totalling 20 uL in A1:D1.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "cycling_and_transformation", + "score": 1, + "evidence": "The comment gives a Golden Gate program of 30 cycles of 37 C/16 C, then 60 C and a 4 C hold. Clean-up elutes in 10 uL, and the eluate goes into the cells with 30 min on ice, 42 C 60 s, 250 uL LB+dextrose, 37 C 60 min recovery and kanamycin plating.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "pcr_setup": 1.0, + "dpni_and_cleanup": 1.0, + "assembly_mix": 1.0, + "cycling_and_transformation": 1.0 + }, + "summary": "The protocol is sound and nearly faithful: the PCR, DpnI, assembly and Golden Gate steps match the paper and the design table, and all tip and safety checks pass. It fails fidelity because it transforms the entire 10 uL eluate and skips the NanoDrop aliquot, where the paper transforms about 5 uL and keeps 5 uL.", + "level": "hard", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "pcr_setup": 0.0625, + "dpni_and_cleanup": 0.0625, + "assembly_mix": 0.0625, + "cycling_and_transformation": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 0.9048, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.4333, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/protocol.py new file mode 100644 index 0000000..58523a5 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/protocol.py @@ -0,0 +1,409 @@ +"""AssemblyTron Golden Gate assembly of four four-fragment chromoprotein +expression plasmids on the OT-2, following: + + Boehm et al., "AssemblyTron: flexible automation of DNA assembly with + Opentrons OT-2 lab robots", Synth. Biol. 2022, doi:10.1093/synbio/ysac032 + +Workflow (paper Methods 2.3/2.4 and Results 3.2, Golden Gate path): + + 1. PCR of 7 fragments (25 uL reactions, Q5 polymerase, 0.1 uM primers, + 0.5 ng linearized plasmid template) set up from a master mix in the + pcr_plate, which stands in for the 100 uL PCR tubes. The OT-2 + thermocycler has no gradient capability, so cycling is done off-deck + on a gradient thermocycler at the AssemblyTron/j5-computed annealing + gradient (manual step). + 2. DpnI digestion of the PCR products to remove the methylated, + template-derived plasmid DNA (paper recipe: 19 uL water + 5 uL + rCutSmart + 1 uL DpnI; 37 C 30 min, 65 C 20 min). + 3. Column clean-up of every fragment (Zymo DNA Clean & Concentrator-5) + to remove residual polymerase, which would otherwise fill in the + BsaI sticky ends (manual step); eluates return to the PCR wells. + 4. Golden Gate assembly: cleaned fragments combined in volumes + proportional to fragment length (roughly equimolar), 10X T4 ligase + buffer, BsaI-HFv2 + T4 ligase mix, water to 20 uL; cycled in a + thermocycler following Engler et al. (manual step). + 5. Clean-up of the assembly reactions, elution in 10 uL water (manual), + and transformation of the whole eluate into 50 uL E. coli TOP10: + 30 min on ice, heat shock 42 C 60 s, recovery 60 min at 37 C with + 250 uL LB + 0.2% dextrose added by the robot. + +Volumes/times not fixed by the paper are chosen below and flagged in +comments. +""" + +from opentrons import protocol_api + +metadata = { + 'protocolName': 'AssemblyTron Golden Gate - four chromoprotein plasmids', + 'author': 'OT-2 protocol from Boehm et al. 2022 (doi:10.1093/synbio/ysac032)', + 'description': 'PCR of 7 fragments, DpnI digest, clean-up, four 20 uL ' + 'Golden Gate assemblies, clean-up, transformation into ' + 'TOP10 competent cells.', + 'apiLevel': '2.13', +} + +# -------------------------------------------------------------------------- +# Design tables (fixed by the j5/AssemblyTron output given in the task) +# -------------------------------------------------------------------------- + +# Fragment number -> (pcr_plate well, fwd primer, rev primer, template well) +FRAGMENTS = { + 1: ('A1', 'A1', 'A2', 'A1'), # tsPurple chromoprotein + 2: ('B1', 'B1', 'B2', 'A1'), # backbone + 3: ('C1', 'C1', 'C2', 'B1'), # YukonOFP chromoprotein + 4: ('D1', 'D1', 'D2', 'C1'), # aeBlue chromoprotein + 5: ('E1', 'E1', 'E2', 'A1'), # backbone / KanR + 6: ('F1', 'F1', 'F2', 'A1'), # backbone + 7: ('G1', 'G1', 'G2', 'D1'), # fuGFP chromoprotein +} + +# assembly_plate well -> {fragment number: volume (uL) of cleaned fragment} +# (volumes proportional to fragment length => roughly equimolar, paper 3.2) +ASSEMBLIES = { + 'A1': {2: 3.0, 5: 2.0, 6: 3.0, 1: 2.0}, # tsPurple construct + 'B1': {2: 3.0, 5: 2.0, 6: 3.0, 3: 2.0}, # YukonOFP construct + 'C1': {2: 3.0, 5: 2.0, 6: 3.0, 4: 2.0}, # aeBlue construct + 'D1': {2: 3.0, 5: 2.0, 6: 3.0, 7: 2.0}, # fuGFP construct +} + +# -------------------------------------------------------------------------- +# Reaction volumes derived from the paper (Methods 2.3) +# -------------------------------------------------------------------------- +# PCR: 25 uL total per reaction, Q5, 0.1 uM primers, 0.5 ng template. +# Per-reaction master mix (buffer + dNTP + polymerase + water) = 19 uL: +MM_WATER = 13.25 # water to 19 uL per reaction +MM_Q5_BUFFER = 5.0 # 5X Q5 reaction buffer -> 1X in 25 uL +MM_DNTP = 0.5 # 10 mM dNTPs -> 200 uM in 25 uL (Q5 standard) +MM_Q5_POL = 0.25 # Q5 polymerase, 0.5 uL/50 uL reaction scaled to 25 uL +# 0.25 uL is below the p20 minimum of 1 uL, which is exactly why the +# polymerase is pipetted through the master mix (8 reactions = 7 fragments +# + 1 spare/dead volume, as staged in tube D1): 8 x 0.25 = 2 uL. +MM_RXN_VOLUME = MM_WATER + MM_Q5_BUFFER + MM_DNTP + MM_Q5_POL # = 19.0 uL +MM_REACTIONS = 8 +PRIMER_VOLUME = 2.5 # 1 uM primer stock -> 0.1 uM in 25 uL (paper) +TEMPLATE_VOLUME = 1.0 # 0.5 ng/uL template -> 0.5 ng per reaction (paper) + +# DpnI digestion of each PCR product, paper's recipe applied to one 25 uL +# PCR: +19 uL water, +5 uL rCutSmart (10X -> 1X in the resulting 50 uL), +# +1 uL DpnI; 37 C 30 min, then 65 C 20 min to inactivate. +DPNI_WATER = 19.0 +DPNI_RCUTSMART = 5.0 +DPNI_ENZYME = 1.0 + +# Column clean-up of fragments (manual). Fragments 2 and 6 are each used at +# 3 uL x 4 assemblies = 12 uL, so the paper's 10 uL elution would be too +# tight; elute in 20 uL water (volume chosen here, not fixed by the paper). +FRAGMENT_ELUTION = 20.0 + +# Golden Gate reaction, 20 uL total (fixed): +GG_T4_BUFFER = 2.0 # 10X T4 DNA ligase buffer -> 1X in 20 uL +GG_ENZYME_MIX = 1.0 # BsaI-HFv2 + T4 ligase premix. The paper does not + # fix the premix volume; 1 uL per 20 uL reaction is + # the standard usage for a combined Golden Gate + # enzyme mix (paper leaves this open -> our choice). +# fragments contribute 10.0 uL (3+2+3+2), so water makes up the balance: +GG_WATER = 20.0 - GG_T4_BUFFER - GG_ENZYME_MIX - 10.0 # = 7.0 uL + +# Assembly clean-up elution: 10 uL water (paper Methods 2.3); the whole +# eluate is transformed into the 50 uL of cells (paper). The paper also +# NanoDrops a spare 5 uL for CFU/ug calculations; skipped here since the +# task ends at transformation (noted at the pause below). +ASSEMBLY_ELUTION = 10.0 + +# Recovery medium: 250 uL LB + 0.2% (w/v) dextrose (paper Methods 2.3). +LB_DEXTROSE_VOLUME = 250.0 + + +def run(protocol: protocol_api.ProtocolContext): + # ------------------------- labware & instruments ---------------------- + tubes_50ml_1 = protocol.load_labware( + 'opentrons_6_tuberack_falcon_50ml_conical', 1, label='tubes_50ml_1') + tubes_1_5ml_1 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 3, label='primer_plate') + template_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='template_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 5, label='pcr_plate') + assembly_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='assembly_plate') + cells_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 7, label='cells_plate') + tubes_15ml_1 = protocol.load_labware( + 'opentrons_15_tuberack_falcon_15ml_conical', 8, label='tubes_15ml_1') + + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + # reagent locations + water = tubes_50ml_1['A1'] # nuclease-free water + q5_buffer = tubes_1_5ml_1['A1'] # 5X Q5 reaction buffer + dntp = tubes_1_5ml_1['B1'] # 10 mM dNTPs + q5_pol = tubes_1_5ml_1['C1'] # Q5 High-Fidelity DNA Polymerase + pcr_mm_tube = tubes_1_5ml_1['D1'] # empty -> PCR master mix made here + rcutsmart = tubes_1_5ml_1['A2'] # rCutSmart Buffer (10X) + dpni = tubes_1_5ml_1['B2'] # DpnI + t4_buffer = tubes_1_5ml_1['C2'] # 10X T4 DNA Ligase Buffer + gg_enzyme = tubes_1_5ml_1['D2'] # BsaI-HFv2 + T4 ligase mix + lb_dextrose = tubes_15ml_1['A1'] # LB + 0.2% (w/v) dextrose + + pcr_wells = [pcr_plate[FRAGMENTS[n][0]] for n in sorted(FRAGMENTS)] + assembly_wells = [assembly_plate[w] for w in sorted(ASSEMBLIES)] + + protocol.comment( + 'AssemblyTron Golden Gate run (Boehm et al. 2022): 7 fragment PCRs ' + '-> DpnI digest -> fragment clean-up -> four 20 uL Golden Gate ' + 'assemblies (tsPurple, YukonOFP, aeBlue, fuGFP) -> assembly ' + 'clean-up -> transformation of TOP10 cells in cells_plate A1:D1. ' + 'Keep enzyme tubes on ice; plates stay on deck between manual steps ' + 'unless stated otherwise.') + + # ====================================================================== + # STEP 1 - PCR master mix (8 reactions: 7 fragments + 1 spare) + # ====================================================================== + protocol.comment( + 'STEP 1: preparing Q5 PCR master mix in tubes_1_5ml_1 D1 for 8 ' + 'reactions (7 PCRs + 1 dead-volume spare): {w:.2f} uL water + ' + '{b:.0f} uL 5X Q5 buffer + {d:.1f} uL 10 mM dNTPs + {p:.1f} uL Q5 ' + 'polymerase = {t:.0f} uL total ({mm:.1f} uL per reaction). ' + 'Polymerase is added through the master mix because 0.25 uL per ' + '25 uL reaction is below the p20 pipetting range.'.format( + w=MM_WATER * MM_REACTIONS, b=MM_Q5_BUFFER * MM_REACTIONS, + d=MM_DNTP * MM_REACTIONS, p=MM_Q5_POL * MM_REACTIONS, + t=MM_RXN_VOLUME * MM_REACTIONS, mm=MM_RXN_VOLUME)) + + p300.transfer(MM_WATER * MM_REACTIONS, water, pcr_mm_tube) + p300.transfer(MM_Q5_BUFFER * MM_REACTIONS, q5_buffer, pcr_mm_tube) + p20.transfer(MM_DNTP * MM_REACTIONS, dntp, pcr_mm_tube) + p20.transfer(MM_Q5_POL * MM_REACTIONS, q5_pol, pcr_mm_tube) + # mix the master mix (last and largest component volume: use p300) + p300.pick_up_tip() + p300.mix(5, 100.0, pcr_mm_tube) + p300.drop_tip() + + # ====================================================================== + # STEP 2 - set up the 7 PCRs in pcr_plate A1:G1 (25 uL each) + # ====================================================================== + protocol.comment( + 'STEP 2: setting up 7 x 25 uL PCRs in pcr_plate A1:G1: {mm:.1f} uL ' + 'master mix + {fp:.1f} uL forward primer + {rp:.1f} uL reverse ' + 'primer (1 uM stocks -> 0.1 uM final, paper 2.3) + {tpl:.1f} uL ' + 'linearized template (0.5 ng/uL -> 0.5 ng, paper 2.3).'.format( + mm=MM_RXN_VOLUME, fp=PRIMER_VOLUME, rp=PRIMER_VOLUME, + tpl=TEMPLATE_VOLUME)) + + # master mix: 19 uL per well is below the p300 20 uL minimum, so the + # p20 (1-20 uL range) dispenses it; a single tip suffices since the tip + # only ever carries master mix. + p20.transfer(MM_RXN_VOLUME, pcr_mm_tube, pcr_wells, new_tip='once') + + for n in sorted(FRAGMENTS): + pcr_well, fwd, rev, tpl = FRAGMENTS[n] + p20.transfer(PRIMER_VOLUME, primer_plate[fwd], + pcr_plate[pcr_well], new_tip='always') + p20.transfer(PRIMER_VOLUME, primer_plate[rev], + pcr_plate[pcr_well], new_tip='always') + + # templates: fresh tip per fragment (template A1 is shared by + # fragments 1, 2, 5 and 6 - no carry-over of DNA between PCRs) + for n in sorted(FRAGMENTS): + pcr_well, fwd, rev, tpl = FRAGMENTS[n] + p20.transfer(TEMPLATE_VOLUME, template_plate[tpl], + pcr_plate[pcr_well], new_tip='always') + + # mix each completed PCR + for well in pcr_wells: + p20.pick_up_tip() + p20.mix(3, 15.0, well) + p20.drop_tip() + + # ------------------ manual: gradient PCR (off deck) ------------------- + protocol.comment( + 'MANUAL STEP - PCR (off-deck gradient thermocycler, e.g. Bio-Rad ' + 'C100; the OT-2 thermocycler has no gradient capability, paper ' + '3.1): seal pcr_plate, transfer it to the gradient thermocycler at ' + 'the block positions/annealing-temperature gradient computed by ' + 'AssemblyTron (each fragment within 0.4 C of its j5 annealing Tm), ' + 'and run: 98 C 30 s initial denaturation; 34 cycles of [98 C 10 s; ' + 'annealing 30 s at the gradient temperature for that fragment; ' + '72 C extension at 20-30 s/kb for Q5 - use 90 s here, sized for ' + 'the longest backbone fragment]; final extension 72 C 5 min; hold ' + '4 C. (Paper uses 34 or 36 cycles; 34 chosen.) Optionally remove ' + 'a small sample of each reaction for agarose gel QC (paper 3.2). ' + 'Return the UNSEALED pcr_plate to deck slot 5 and resume.') + protocol.pause('Run the gradient PCR off-deck, then return pcr_plate to ' + 'slot 5 and resume.') + + # ====================================================================== + # STEP 3 - DpnI digestion of the PCR products (paper 2.3 recipe) + # ====================================================================== + protocol.comment( + 'STEP 3: DpnI digest to destroy the methylated plasmid template ' + 'carried over from the PCRs: adding {w:.0f} uL water + {b:.0f} uL ' + 'rCutSmart (10X -> 1X in the resulting 50 uL) + {e:.0f} uL DpnI to ' + 'each 25 uL PCR in pcr_plate A1:G1.'.format( + w=DPNI_WATER, b=DPNI_RCUTSMART, e=DPNI_ENZYME)) + + p20.transfer(DPNI_WATER, water, pcr_wells, new_tip='once') + p20.transfer(DPNI_RCUTSMART, rcutsmart, pcr_wells, new_tip='once') + p20.transfer(DPNI_ENZYME, dpni, pcr_wells, new_tip='once') + + for well in pcr_wells: + p300.pick_up_tip() + p300.mix(3, 30.0, well) + p300.drop_tip() + + # ---------------- manual: DpnI incubation (off deck) ------------------ + protocol.comment( + 'MANUAL STEP - DpnI digestion (paper 2.3): seal pcr_plate and ' + 'incubate in a thermocycler at 37 C for 30 min, then deactivate ' + 'DpnI at 65 C for 20 min. Unseal and return the plate to deck ' + 'slot 5.') + protocol.pause('Incubate pcr_plate: 37 C 30 min, then 65 C 20 min; ' + 'return to slot 5 and resume.') + + # ---------------- manual: fragment clean-up (columns) ----------------- + protocol.comment( + 'MANUAL STEP - fragment clean-up (paper 3.2: residual polymerase ' + 'fills in the BsaI sticky ends and must be removed before ' + 'assembly): clean and concentrate EACH of the 7 digested fragments ' + '(50 uL each in pcr_plate A1:G1) on a Zymo DNA Clean & ' + 'Concentrator-5 column (add 5 volumes DNA Binding Buffer, load, ' + 'wash twice with 200 uL DNA Wash Buffer, then elute with {el:.0f} ' + 'uL nuclease-free water). Elute in {el:.0f} uL rather than the ' + 'paper\'s 10 uL final-assembly elution because shared fragments ' + 'need up to 12 uL (3 uL x 4 assemblies) plus pipetting dead ' + 'volume - our choice where the paper leaves fragment elution open. ' + 'Return each eluate to its ORIGINAL pcr_plate well (fragment 1 -> ' + 'A1, 2 -> B1, 3 -> C1, 4 -> D1, 5 -> E1, 6 -> F1, 7 -> G1).'.format( + el=FRAGMENT_ELUTION)) + protocol.pause('Column-purify the 7 fragments, elute each in 20 uL ' + 'water back into its pcr_plate well, and resume.') + + # ====================================================================== + # STEP 4 - Golden Gate assembly set-up in assembly_plate A1:D1 (20 uL) + # ====================================================================== + protocol.comment( + 'STEP 4: setting up four 20 uL Golden Gate reactions in ' + 'assembly_plate A1 (tsPurple), B1 (YukonOFP), C1 (aeBlue), D1 ' + '(fuGFP): {w:.1f} uL water + {b:.1f} uL 10X T4 ligase buffer + ' + 'cleaned fragments in length-proportional volumes (3+2+3+2 = ' + '10 uL; backbone fragments 2/5/6 shared by all four constructs, ' + 'plus one chromoprotein fragment each) + {e:.1f} uL BsaI-HFv2/T4 ' + 'ligase mix added last to start the reactions.'.format( + w=GG_WATER, b=GG_T4_BUFFER, e=GG_ENZYME_MIX)) + + p20.transfer(GG_WATER, water, assembly_wells, new_tip='once') + p20.transfer(GG_T4_BUFFER, t4_buffer, assembly_wells, new_tip='once') + + # shared backbone fragments: one tip per fragment source is safe - the + # tip only ever contacts that one fragment plus wells that all receive + # it. (fragment 2: 3 uL, fragment 5: 2 uL, fragment 6: 3 uL per well) + for frag_n in (2, 5, 6): + src = pcr_plate[FRAGMENTS[frag_n][0]] + vol = ASSEMBLIES['A1'][frag_n] # identical across the 4 assemblies + p20.transfer(vol, src, assembly_wells, new_tip='once') + + # chromoprotein fragment: different source per assembly -> fresh tip + # each time; volumes are identical (2 uL) across assemblies. + for asm_well, frags in ASSEMBLIES.items(): + chromo = [n for n in frags if n not in (2, 5, 6)][0] + p20.transfer(frags[chromo], pcr_plate[FRAGMENTS[chromo][0]], + assembly_plate[asm_well], new_tip='always') + + # enzyme mix last (starts the digestion/ligation cycling), then mix; + # fresh tip per well since mix_after re-enters each reaction + p20.transfer(GG_ENZYME_MIX, gg_enzyme, assembly_wells, + new_tip='always', mix_after=(3, 15.0)) + + # ---------------- manual: Golden Gate cycling ------------------------- + protocol.comment( + 'MANUAL STEP - Golden Gate thermocycling (paper 3.2, based on ' + 'Engler et al.): seal assembly_plate A1:D1 and seat it on the ' + 'Opentrons thermocycler module (or move it to any thermocycler if ' + 'no module is available - the protocol pauses either way). Run ' + 'with a heated lid: 30 cycles of [37 C 5 min (BsaI digestion); ' + '16 C 5 min (T4 ligation)]; final 37 C 15 min to drive assembly ' + 'to completion; 60 C 5 min to heat-inactivate; hold 4 C. (Cycle ' + 'count/lengths follow the Engler one-pot protocol the paper ' + 'cites; the paper does not print the program.) Return/seal the ' + 'plate in deck slot 6 when done.') + protocol.pause('Run the Golden Gate cycling program on assembly_plate, ' + 'then resume.') + + # ---------------- manual: assembly clean-up (columns) ----------------- + protocol.comment( + 'MANUAL STEP - assembly clean-up (paper 2.3): clean and ' + 'concentrate each 20 uL Golden Gate reaction on a Zymo DNA Clean & ' + 'Concentrator-5 column and elute with {el:.0f} uL molecular-grade ' + 'water. Return each eluate to its assembly_plate well (A1:D1). ' + 'The paper NanoDrops a spare 5 uL of eluate to compute CFU/ug ' + 'transformation efficiencies; skipped here since this run ends at ' + 'transformation and the whole 10 uL eluate is transformed.'.format( + el=ASSEMBLY_ELUTION)) + protocol.pause('Column-purify the 4 assemblies, elute each in 10 uL ' + 'water back into assembly_plate A1:D1, and resume.') + + # ====================================================================== + # STEP 5 - transformation into E. coli TOP10 (cells_plate A1:D1) + # ====================================================================== + protocol.comment( + 'STEP 5: adding each {el:.0f} uL cleaned assembly eluate to 50 uL ' + 'of TOP10 chemically competent cells (Hanahan method) in ' + 'cells_plate A1 (tsPurple), B1 (YukonOFP), C1 (aeBlue), D1 ' + '(fuGFP), with a gentle 2x mix.'.format(el=ASSEMBLY_ELUTION)) + + p20.transfer(ASSEMBLY_ELUTION, + [assembly_plate[w] for w in sorted(ASSEMBLIES)], + [cells_plate[w] for w in sorted(ASSEMBLIES)], + new_tip='always', mix_after=(2, 10.0)) + + # ---------------- manual: ice incubation + heat shock ----------------- + protocol.comment( + 'MANUAL STEP - transformation (paper 2.3): incubate the cells on ' + 'ice for 30 min, heat-shock at 42 C for exactly 60 s (heat block ' + 'or water bath; cannot be done on-deck), then return the plate to ' + 'ice briefly. Bring cells_plate back to deck slot 7 and resume so ' + 'the robot can add recovery medium.') + protocol.pause('Incubate 30 min on ice, heat shock 42 C 60 s, return ' + 'cells_plate to slot 7 and resume.') + + # recovery medium, added by the robot (paper 2.3: 250 uL LB + 0.2% + # dextrose; 250 uL fits the p300 and keeps the 360 uL wells < capacity) + protocol.comment( + 'Adding {v:.0f} uL LB + 0.2% (w/v) dextrose recovery medium from ' + 'tubes_15ml_1 A1 to each transformation well.'.format( + v=LB_DEXTROSE_VOLUME)) + p300.transfer(LB_DEXTROSE_VOLUME, lb_dextrose, + [cells_plate[w] for w in sorted(ASSEMBLIES)], + new_tip='always', mix_after=(3, 200.0)) + + # ---------------- manual: outgrowth recovery -------------------------- + protocol.comment( + 'MANUAL STEP - recovery (paper 2.3): incubate the transformation ' + 'mixes at 37 C for 60 min with shaking/aeration to allow ' + 'kanamycin-resistance expression.') + protocol.pause('Recover cells at 37 C for 60 min, then resume to end ' + 'the robot run.') + + # ---------------- manual: plating (end of robot work) ----------------- + protocol.comment( + 'MANUAL STEP - plating and scoring (paper 2.3/3.2): plate 50-200 ' + 'uL of each recovery (neat or as a 10x dilution, depending on ' + 'expected efficiency; the paper saw 300-800 colonies per assembly) ' + 'onto LB agar with kanamycin 50 ug/mL - all four constructs carry ' + 'the KanR fragment 5. Incubate at 37 C overnight, then count ' + 'colonies and score chromoprotein colour: A1 purple (tsPurple), ' + 'B1 orange (YukonOFP), C1 blue (aeBlue), D1 green (fuGFP); >98% ' + 'of colonies are expected to show the correct colour.') + protocol.comment( + 'Robot work complete. Intermediates left on deck: spare PCR master ' + 'mix in tubes_1_5ml_1 D1, unused primer/template stocks, and ' + 'fragment leftovers in pcr_plate A1:G1 for re-use or archiving.') diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/reward.json new file mode 100644 index 0000000..810616a --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.75, "sim_pass": 1.0, "checks_frac": 0.9048, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.4333, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_pcr_setup": 1.0, "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 1.0, "rubric_cycling_and_transformation": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/trial.json new file mode 100644 index 0000000..1292406 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/trial.json @@ -0,0 +1,31 @@ +{ + "task": "golden-gate-assembly-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 0.9048, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.4333, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + }, + "exception": null, + "cost_usd": 0.671698, + "input_tokens": 596708, + "output_tokens": 47617, + "started_at": "2026-10-07T18:21:21.341883Z", + "finished_at": "2026-10-07T18:44:49.101158Z", + "cache_tokens": 461383, + "claude_code_cost_usd": 2.20125525 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/grader.json new file mode 100644 index 0000000..0881b9a --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/grader.json @@ -0,0 +1,236 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 549 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:water", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dntp", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_pol", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_mm", + "pass": false, + "detail": "expected {'D1': 19.0}, got {'D1': 12.0}" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:template_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": false, + "detail": "expected {'A1': 48.0, 'B1': 38.0, 'C1': 48.0, 'D1': 48.0, 'E1': 42.0, 'F1': 38.0, 'G1': 48.0}, got {'A1': 48.0, 'B1': 37.0, 'C1': 48.0, 'D1': 48.0, 'E1': 41.0, 'F1': 37.0, 'G1': 48.0}" + }, + { + "name": "end_state:rcutsmart", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dpni", + "pass": true, + "detail": "" + }, + { + "name": "end_state:t4_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:gg_enzyme", + "pass": true, + "detail": "" + }, + { + "name": "end_state:assembly_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:cells_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:lb_dextrose", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "Valid apiLevel 2.13 and metadata. Every volume is within its pipette's range (p20 for \u226420 \u00b5L, p300 for 40, 106, 250 \u00b5L and the 30 \u00b5L mixes), and no well is overfilled (cells wells hold 305 \u00b5L of 360).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "The tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks all passed. Fresh tips are used per primer, template and fragment transfer, and shared reagents are dispensed with distribute using one tip.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 0, + "evidence": "The pcr_mm end_state check failed (D1 holds 12 \u00b5L instead of 19 \u00b5L). p20.distribute(19, ...) used a disposal volume that discarded master mix. Step 24 also mixes at 15 \u00b5L, which differs from the reference.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "pcr_setup", + "score": 0, + "evidence": "Each PCR well receives the correct 19 \u00b5L mix, 2.5 \u00b5L primers and 1 \u00b5L template. However, the end_state:pcr_mm check failed (12 vs 19 \u00b5L) because p20.distribute(19, pcr_mm, pcr_wells) discarded master mix.", + "votes": [ + 0, + 1, + 0 + ] + }, + { + "id": "dpni_and_cleanup", + "score": 1, + "evidence": "Steps 12-14 add 19 \u00b5L water, 5 \u00b5L rCutSmart and 1 \u00b5L DpnI. The 3x30 \u00b5L mix, incubation comment and Zymo cleanup comment are in order, and the DpnI volumes match.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "assembly_mix", + "score": 0, + "evidence": "The end_state:pcr_plate check failed (B1, E1 and F1 are 1 \u00b5L low). p20.distribute of fragments 2, 5 and 6 with a disposal volume wasted fragment, so the assembly step does not match the task's volumes.", + "votes": [ + 0, + 1, + 0 + ] + }, + { + "id": "cycling_and_transformation", + "score": 1, + "evidence": "Step 25 records the Golden Gate program (30 cycles of 37/16\u00b0C, 60\u00b0C, 4\u00b0C hold) via comment. Step 27 transfers 5 \u00b5L to the cells, followed by the ice/heat-shock comment and 250 \u00b5L LB. The cells_plate end_state check passed.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 0.0, + "pcr_setup": 0.0, + "dpni_and_cleanup": 1.0, + "assembly_mix": 0.0, + "cycling_and_transformation": 1.0 + }, + "summary": "The protocol simulates cleanly with correct tip handling, and the manual steps are recorded as comments. However, p20.distribute() used a disposal volume that discarded master mix and fragment volume, which failed the pcr_mm and pcr_plate end-state checks. That fails the PCR setup, assembly mix and fidelity items.", + "level": "easy", + "score": 0.625, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "pcr_setup": 0.0625, + "dpni_and_cleanup": 0.0625, + "assembly_mix": 0.0625, + "cycling_and_transformation": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [ + "end_state:pcr_mm", + "end_state:pcr_plate" + ], + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9048, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.4333, + "judge_score": 0.625, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 0.0, + "rubric_pcr_setup": 0.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 0.0, + "rubric_cycling_and_transformation": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/protocol.py new file mode 100644 index 0000000..0c65382 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/protocol.py @@ -0,0 +1,269 @@ +from opentrons import protocol_api + +metadata = { + 'protocolName': 'AssemblyTron Golden Gate assembly of four chromoprotein ' + 'expression plasmids (4 fragments each)', + 'author': 'Claude', + 'description': 'Golden Gate assembly on the OT-2: Q5 PCR of fragments 1-7, ' + 'DpnI digest of template, fragment cleanup, Golden Gate ' + 'assembly (BsaI-HFv2 + T4 ligase), cleanup and ' + 'transformation into E. coli TOP10.', + 'apiLevel': '2.13' +} + + +def run(protocol: protocol_api.ProtocolContext): + + # ------------------------------------------------------------------ + # Labware + # ------------------------------------------------------------------ + tubes_50ml_1 = protocol.load_labware( + 'opentrons_6_tuberack_falcon_50ml_conical', 1, label='tubes_50ml_1') + tubes_1_5ml_1 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 3, label='primer_plate') + template_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='template_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 5, label='pcr_plate') + assembly_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='assembly_plate') + cells_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 7, label='cells_plate') + tubes_15ml_1 = protocol.load_labware( + 'opentrons_15_tuberack_falcon_15ml_conical', 8, label='tubes_15ml_1') + + # ------------------------------------------------------------------ + # Pipettes and tips + # ------------------------------------------------------------------ + tiprack_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tiprack_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tiprack_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tiprack_300]) + + # ------------------------------------------------------------------ + # Locations + # ------------------------------------------------------------------ + water = tubes_50ml_1['A1'] + q5_buffer = tubes_1_5ml_1['A1'] + dntp = tubes_1_5ml_1['B1'] + q5_pol = tubes_1_5ml_1['C1'] + pcr_mm = tubes_1_5ml_1['D1'] # empty at start -> PCR master mix + rcutsmart = tubes_1_5ml_1['A2'] + dpni = tubes_1_5ml_1['B2'] + t4_buffer = tubes_1_5ml_1['C2'] + gg_enzyme = tubes_1_5ml_1['D2'] + lb_dextrose = tubes_15ml_1['A1'] + + # PCR wells (column 1): fragments 1-7 in A1:G1 + frag_names = ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1'] + pcr_wells = [pcr_plate[w] for w in frag_names] + + fwd_primers = [primer_plate[w] for w in frag_names] + rev_primers = [primer_plate['%s2' % r] for r in 'ABCDEFG'] + + # Template for each fragment 1-7 (template_plate wells) + # frag1=tsPurple(A1), frag2=backbone(A1), frag3=YukonOFP(B1), + # frag4=aeBlue(C1), frag5=backbone/KanR(A1), frag6=backbone(A1), + # frag7=fuGFP(D1) + template_wells = ['A1', 'A1', 'B1', 'C1', 'A1', 'A1', 'D1'] + templates = [template_plate[w] for w in template_wells] + + assembly_wells = [assembly_plate[w] for w in ['A1', 'B1', 'C1', 'D1']] + cells_wells = [cells_plate[w] for w in ['A1', 'B1', 'C1', 'D1']] + + # ================================================================== + # PCR master mix (for 8 reactions of 19 uL: water 106, 5X Q5 buffer + # 40, 10 mM dNTPs 4, Q5 polymerase 2 = 152 uL) + # ================================================================== + protocol.comment('Step 1: Adding 106 uL nuclease-free water to the PCR ' + 'master mix tube (tubes_1_5ml_1 D1).') + p300.transfer(106, water, pcr_mm, new_tip='always') + + protocol.comment('Step 2: Adding 40 uL 5X Q5 reaction buffer to the PCR ' + 'master mix tube.') + p300.transfer(40, q5_buffer, pcr_mm, new_tip='always') + + protocol.comment('Step 3: Adding 4 uL 10 mM dNTPs to the PCR master mix ' + 'tube.') + p20.transfer(4, dntp, pcr_mm, new_tip='always') + + protocol.comment('Step 4: Adding 2 uL Q5 High-Fidelity DNA Polymerase to ' + 'the PCR master mix tube.') + p20.transfer(2, q5_pol, pcr_mm, new_tip='always') + + protocol.comment('Step 5: Mixing the PCR master mix (5x at 100 uL).') + p300.pick_up_tip() + p300.mix(5, 100, pcr_mm) + p300.drop_tip() + + # ================================================================== + # PCR setup in pcr_plate A1:G1 (25 uL each) + # ================================================================== + protocol.comment('Step 6: Distributing 19 uL PCR master mix to pcr_plate ' + 'A1:G1 (fragments 1-7).') + p20.distribute(19, pcr_mm, pcr_wells) + + protocol.comment('Step 7: Adding 2.5 uL of each forward primer (1 uM) to ' + 'the corresponding PCR well.') + for src, dst in zip(fwd_primers, pcr_wells): + p20.transfer(2.5, src, dst, new_tip='always') + + protocol.comment('Step 8: Adding 2.5 uL of each reverse primer (1 uM) to ' + 'the corresponding PCR well.') + for src, dst in zip(rev_primers, pcr_wells): + p20.transfer(2.5, src, dst, new_tip='always') + + protocol.comment('Step 9: Adding 1 uL of each linearized template ' + '(0.5 ng/uL, 0.5 ng) to the corresponding PCR well.') + for src, dst in zip(templates, pcr_wells): + p20.transfer(1, src, dst, new_tip='always') + + protocol.comment('Step 10: Mixing each PCR (3x at 15 uL).') + for well in pcr_wells: + p20.pick_up_tip() + p20.mix(3, 15, well) + p20.drop_tip() + + # ------------------------------------------------------------------ + # Gradient PCR (manual, off-robot) + # ------------------------------------------------------------------ + protocol.comment( + 'Step 11 (manual): Seal pcr_plate (or move the PCR tubes) and ' + 'transfer to the Bio-Rad C100 gradient thermocycler. Run: 98C 30 s; ' + '34 cycles of [98C 10 s, annealing 30 s at the AssemblyTron/j5 ' + 'optimal-gradient temperature for each fragment, 72C extension at ' + 'the time set by AssemblyTron (~20-30 s/kb)]; final extension 72C ' + '5 min; hold 4C. Take a sample of each reaction for gel ' + 'electrophoresis, then return the reactions to the OT-2.') + + # ================================================================== + # DpnI digestion of residual template (50 uL total per reaction) + # ================================================================== + protocol.comment('Step 12: Adding 19 uL nuclease-free water to each PCR ' + '(A1:G1) for DpnI digestion.') + p20.distribute(19, water, pcr_wells) + + protocol.comment('Step 13: Adding 5 uL rCutSmart Buffer to each PCR.') + p20.distribute(5, rcutsmart, pcr_wells) + + protocol.comment('Step 14: Adding 1 uL DpnI to each PCR.') + p20.distribute(1, dpni, pcr_wells) + + protocol.comment('Step 15: Mixing each DpnI digestion (3x at 30 uL).') + for well in pcr_wells: + p300.pick_up_tip() + p300.mix(3, 30, well) + p300.drop_tip() + + protocol.comment( + 'Step 16 (manual): Incubate pcr_plate A1:G1 at 37C for 30 min, then ' + '65C for 20 min (DpnI inactivation) on the thermocycler block.') + + # ------------------------------------------------------------------ + # Fragment clean-up (manual) + # ------------------------------------------------------------------ + protocol.comment( + 'Step 17 (manual): PAUSE. Clean and concentrate each of the 7 ' + 'fragments with a Zymo DNA Clean & Concentrator-5 column: add 5:1 ' + 'DNA Binding Buffer to sample (250 uL per 50 uL), spin 30 s, wash ' + '2 x 200 uL DNA Wash Buffer (30 s spins), elute in 20 uL water ' + 'after 1 min at room temperature (30 s spin). Return the eluted ' + 'fragments (20 uL each) to pcr_plate A1:G1 (fragments 1-7) and ' + 'resume the protocol.') + + # ================================================================== + # Golden Gate assembly setup in assembly_plate A1:D1 (20 uL each) + # Assemblies: A1 = tsPurple (purple), B1 = YukonOFP (orange), + # C1 = aeBlue (blue), D1 = fuGFP (green) + # Each = fragment 2 (backbone) + fragment 5 (backbone/KanR) + + # fragment 6 (backbone) + one chromoprotein fragment (1,3,4,7) + # ================================================================== + protocol.comment('Step 18: Adding 7 uL nuclease-free water to ' + 'assembly_plate A1:D1.') + p20.distribute(7, water, assembly_wells) + + protocol.comment('Step 19: Adding 2 uL 10X T4 DNA Ligase Buffer to each ' + 'assembly.') + p20.distribute(2, t4_buffer, assembly_wells) + + protocol.comment('Step 20: Adding 3 uL of fragment 2 (backbone) to each ' + 'assembly.') + p20.distribute(3, pcr_plate['B1'], assembly_wells) + + protocol.comment('Step 21: Adding 2 uL of fragment 5 (backbone/KanR) to ' + 'each assembly.') + p20.distribute(2, pcr_plate['E1'], assembly_wells) + + protocol.comment('Step 22: Adding 3 uL of fragment 6 (backbone) to each ' + 'assembly.') + p20.distribute(3, pcr_plate['F1'], assembly_wells) + + protocol.comment('Step 23: Adding 2 uL of each chromoprotein fragment ' + '(1=tsPurple->A1, 3=YukonOFP->B1, 4=aeBlue->C1, ' + '7=fuGFP->D1) to the corresponding assembly.') + chromo_sources = [pcr_plate[w] for w in ['A1', 'C1', 'D1', 'G1']] + for src, dst in zip(chromo_sources, assembly_wells): + p20.transfer(2, src, dst, new_tip='always') + + protocol.comment('Step 24: Adding 1 uL Golden Gate Enzyme Mix ' + '(BsaI-HFv2 + T4 ligase) to each assembly and mixing ' + '5x.') + for dst in assembly_wells: + p20.transfer(1, gg_enzyme, dst, mix_after=(5, 15), new_tip='always') + + # ------------------------------------------------------------------ + # Golden Gate thermocycling (manual / Opentrons thermocycler module) + # ------------------------------------------------------------------ + protocol.comment( + 'Step 25 (manual): Run the Golden Gate program on assembly_plate in ' + 'the Opentrons thermocycler module: 30 cycles of 37C 5 min then 16C ' + '5 min; then 60C 5 min; hold at 4C. Lid about 85C. If no module is ' + 'available, pause and move the reactions to another thermocycler.') + + protocol.comment( + 'Step 26 (manual): Clean and concentrate each assembly with a Zymo ' + 'DNA Clean & Concentrator-5 column (5:1 DNA Binding Buffer to ' + 'sample, wash 2 x 200 uL DNA Wash Buffer) and elute in 10 uL ' + 'molecular grade water. Return the eluates (10 uL each) to ' + 'assembly_plate A1:D1.') + + # ================================================================== + # Transformation into E. coli TOP10 + # ================================================================== + protocol.comment('Step 27: Adding 5 uL of each purified Golden Gate ' + 'assembly to the corresponding well of competent ' + 'TOP10 cells (cells_plate A1:D1).') + for src, dst in zip(assembly_wells, cells_wells): + p20.transfer(5, src, dst, new_tip='always') + + protocol.comment( + 'Step 28 (manual): Measure DNA concentration of the remaining 5 uL ' + 'of each eluate with a NanoDrop-2000c for CFU/ug calculation.') + + protocol.comment( + 'Step 29 (manual): Incubate cells_plate A1:D1 for 30 min on ice, ' + 'heat shock at 42C for 60 s, then return to ice for about 2 min.') + + protocol.comment('Step 30: Adding 250 uL LB + 0.2% (w/v) dextrose to ' + 'each transformation for outgrowth.') + p300.distribute(250, lb_dextrose, cells_wells) + + protocol.comment( + 'Step 31 (manual): Recover the cells at 37C for 60 min with shaking ' + '(about 250 rpm).') + + protocol.comment( + 'Step 32 (manual): Plate 50-200 uL of each recovery (neat or a 10x ' + 'dilution, depending on predicted efficiency) onto separate LB agar ' + 'plates containing kanamycin 50 ug/mL; incubate at 37C overnight.') + + protocol.comment( + 'Step 33 (manual): Count colonies per plate and score the fraction ' + 'showing the expected chromoprotein colour (A1 purple, B1 orange, ' + 'C1 blue, D1 green); report CFU/ug of DNA plated.') diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/reward.json new file mode 100644 index 0000000..da56884 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.9048, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.4333, "judge_score": 0.625, "critical_fail": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 0.0, "rubric_pcr_setup": 0.0, "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 0.0, "rubric_cycling_and_transformation": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/trial.json new file mode 100644 index 0000000..daa39ca --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/golden-gate-assembly/trial.json @@ -0,0 +1,31 @@ +{ + "task": "golden-gate-assembly", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9048, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.4333, + "judge_score": 0.625, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 0.0, + "rubric_pcr_setup": 0.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 0.0, + "rubric_cycling_and_transformation": 1.0 + }, + "exception": null, + "cost_usd": 0.199604, + "input_tokens": 211070, + "output_tokens": 11699, + "started_at": "2026-10-07T18:21:21.437690Z", + "finished_at": "2026-10-07T18:29:26.269140Z", + "cache_tokens": 167274, + "claude_code_cost_usd": 0.6497919999999999 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/grader.json new file mode 100644 index 0000000..c3f4ac2 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/grader.json @@ -0,0 +1,234 @@ +{ + "variant": "reference", + "judge_model": "claude-sonnet-5-5", + "suspicious_tokens": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1450 + }, + "checks": { + "checks": [ + { + "name": "48_samples_to_odd_columns", + "pass": true, + "detail": "48 tubes -> 48 wells, odd columns only: True" + }, + { + "name": "step_order", + "pass": true, + "detail": "expected per well: beads > isopropanol > sample > remove > ethanol > remove > ethanol > remove > elution > recover" + }, + { + "name": "binding_volumes_40_250_250", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "mix_5x_after_sample", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "incubation_5min_before_magnet", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_4min_before_first_removal", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "supernatant_removed_each_step", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_engaged_for_all_removals", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "two_500ul_ethanol_washes", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "air_dry_4min", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_100ul", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_off_magnet_then_90s_on", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "recover_70_100ul_one_well_each", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "distinct_elution_wells", + "pass": true, + "detail": "48 distinct elution wells" + }, + { + "name": "elution_plate_4C_before_recovery", + "pass": true, + "detail": "temperatures set: [(0.0, 4.0)]" + }, + { + "name": "fresh_tip_per_sample_no_cross_contact", + "pass": true, + "detail": "48 distinct tips for 48 sample transfers; violations: []" + }, + { + "name": "reservoir_columns_within_15ml", + "pass": true, + "detail": "uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 12000, 10: 12000, 11: 12000, 12: 12000" + } + ], + "checks_passed": 17, + "checks_total": 17, + "total_delay_min": 15.0, + "example_well": "A1", + "example_sequence": [ + "beads 40uL", + "isopropanol 250uL", + "sample 250uL", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "elution 100uL", + "recover 90uL (magnet)" + ] + }, + "critical_failures": [], + "judge": { + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.11 is valid. The p300 multi splits 250 uL transfers into 125 uL pieces, p1000 handles 250 uL, volumes in the reservoir are within capacity (reservoir check passed), and the simulator ran cleanly.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips for the 48 samples. Every pick_up_tip is paired with drop_tip, and the 34 multichannel pickups fit within the 36 available tip columns.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "Each removal takes only 2x250 = 500 uL from 540 uL wells, so about 40 uL stays after every step, and the comment 'remove supernatant (~540 uL)' is inaccurate. This leaves residual ethanol before elution.", + "votes": [ + 1, + 0, + 0 + ] + }, + { + "id": "sample_handling", + "score": 1, + "evidence": "Checks 48_samples_to_odd_columns and distinct_elution_wells passed. The code maps 48 tubes to odd-column wells and recovers each eluate into the same column of the elution plate.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding_and_separation", + "score": 1, + "evidence": "Checks binding_volumes_40_250_250, mix_5x_after_sample, incubation_5min_before_magnet, magnet_4min_before_first_removal and magnet_engaged_for_all_removals all passed. The code pipettes beads, then isopropanol, then sample, and removes the supernatant to waste with the magnet engaged.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "washes_and_drying", + "score": 1, + "evidence": "Checks two_500ul_ethanol_washes, supernatant_removed_each_step and air_dry_4min passed. Each wash removal runs with the magnet engaged, and the code delays 4 min before mag_mod.disengage().", + "votes": [ + 1, + 0, + 1 + ] + }, + { + "id": "elution_recovery", + "score": 1, + "evidence": "Checks elution_100ul, elution_off_magnet_then_90s_on, recover_70_100ul_one_well_each and elution_plate_4C_before_recovery passed. The code adds 100 uL off-magnet, mixes, engages the magnet for 90 s, then moves 90 uL to the 4 C plate.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "sample_handling": 1.0, + "binding_and_separation": 1.0, + "washes_and_drying": 1.0, + "elution_recovery": 1.0 + }, + "summary": "The protocol passed all 17 measured run-log checks and handles tips correctly, using a fresh p1000 tip for each of the 48 samples and dropping every tip. I scored it 1 on every item, including the all-or-nothing fidelity item, whose only blemishes are minor: a \\\"~540 uL\\\" comment against 500 uL actually removed, and the 30 s elution wait coming after all columns are mixed.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.11 is valid. The p300 multi splits 250 uL transfers into 125 uL pieces, p1000 handles 250 uL, volumes in the reservoir are within capacity (reservoir check passed), and the simulator ran cleanly.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips for the 48 samples. Every pick_up_tip is paired with drop_tip, and the 34 multichannel pickups fit within the 36 available tip columns.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"All 17 run-log checks passed: volumes 40/250/250, mix 5x, 5 min incubation, 4 min magnet, two 500 uL ethanol washes, 4 min dry, 100 uL elution, 90 s magnet, and recovery of 90 uL at 4 C. The metadata cites the paper correctly; the only blemishes are the \\\"~540 uL\\\" comment against 500 uL actually removed and the 30 s wait coming after all elution mixing.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks 48_samples_to_odd_columns and distinct_elution_wells passed. The code maps 48 tubes to odd-column wells and recovers each eluate into the same column of the elution plate.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks binding_volumes_40_250_250, mix_5x_after_sample, incubation_5min_before_magnet, magnet_4min_before_first_removal and magnet_engaged_for_all_removals all passed. The code pipettes beads, then isopropanol, then sample, and removes the supernatant to waste with the magnet engaged.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Checks two_500ul_ethanol_washes, supernatant_removed_each_step and air_dry_4min passed. Each wash removal runs with the magnet engaged, and the code delays 4 min before mag_mod.disengage().\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"Checks elution_100ul, elution_off_magnet_then_90s_on, recover_70_100ul_one_well_each and elution_plate_4C_before_recovery passed. The code adds 100 uL off-magnet, mixes, engages the magnet for 90 s, then moves 90 uL to the 4 C plate.\"}], \"summary\": \"The protocol passed all 17 measured run-log checks and handles tips correctly, using a fresh p1000 tip for each of the 48 samples and dropping every tip. I scored it 1 on every item, including the all-or-nothing fidelity item, whose only blemishes are minor: a \\\\\\\"~540 uL\\\\\\\" comment against 500 uL actually removed, and the 30 s elution wait coming after all columns are mixed.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.11, p300 multi volumes split into 125 uL chunks within the 200 uL tips, p1000 handles 250 uL and 300 uL mixes, reservoir columns stay under 15 mL (check passed).\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips. Every tip is dropped, with a fresh tip per column for removals and elution. Shared tips are used only for the same reagent, dispensed above the liquid.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"Each removal takes only 2x250 = 500 uL from 540 uL wells, so about 40 uL stays after every step, and the comment 'remove supernatant (~540 uL)' is inaccurate. This leaves residual ethanol before elution.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks 48_samples_to_odd_columns and distinct_elution_wells passed. One fresh p1000 tip moves each sample into its own odd-column well.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks passed for volumes 40/250/250, 5x mix, 5 min incubation, 4 min magnet and removal with magnet on. The removal leaves about 40 uL, but that is minor here.\"}, {\"id\": \"washes_and_drying\", \"score\": 0, \"evidence\": \"Each wash removal aspirates only 2x250 = 500 uL from a 540 uL well, so about 40 uL of ethanol remains after wash 2. The washes are not fully removed before the 4 min dry.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"100 uL elution is added with the magnet disengaged and mixed 10x. The magnet is engaged after 30 s and held 90 s, then 90 uL is moved to distinct wells of the 4 C plate (checks passed).\"}], \"summary\": \"The protocol passes all measured checks and is sound on tips and deck use. However, the supernatant and wash removals take only 500 uL from 540 uL wells, leaving about 40 uL (residual ethanol after wash 2), and the comment says about 540 uL is removed, so the washes item and the all-or-nothing fidelity item fail.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.11, p1000 handles 250 \\u00b5L samples, p300 multi handles \\u2264250 \\u00b5L transfers, waste wells hold at most 1500 \\u00b5L, and the simulation passed with no volume or capacity problems.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips. Every pipetting step has a tip. Each aspirate/dispense-to-waste or elution step uses a fresh tip per column. Every tip is dropped.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"Each removal takes only 2\\u00d7250=500 \\u00b5L from a 540 \\u00b5L binding mix, so about 40 \\u00b5L is left behind. The comment 'remove supernatant (~540 uL)' is therefore inaccurate. Residual liquid carries into the washes and elution, so this is not a fully faithful implementation.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check '48_samples_to_odd_columns' passed, with 250 \\u00b5L moved per sample into odd-column wells. Check 'distinct_elution_wells' confirms 48 separate elution wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks passed for volumes 40/250/250, 5\\u00d7 mix, 5 min incubation, 4 min magnet, and supernatant removal with the magnet engaged. The code does beads, then isopropanol, then sample with mix_after=(5,300).\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Checks 'two_500ul_ethanol_washes', 'magnet_engaged_for_all_removals' and 'air_dry_4min' passed. Each 500 \\u00b5L wash is added and 500 \\u00b5L is removed with the magnet on, then a 4 min delay, then disengage.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"Checks passed for 100 \\u00b5L elution off-magnet, 90 s magnet re-engagement, 90 \\u00b5L recovery, distinct wells and 4 \\u00b0C set. The code disengages, adds buffer and mixes, waits 30 s, engages for 90 s, then transfers to the elution plate.\"}], \"summary\": \"The protocol passes all 17 measured checks, uses fresh tips correctly, and performs the binding, wash, drying and elution steps with the right volumes and timing. It fails fidelity because each removal takes 500 \\u00b5L from a 540 \\u00b5L mix, leaving about 40 \\u00b5L behind, while a comment claims ~540 \\u00b5L is removed.\"}]", + "level": "hard", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "sample_handling": 0.0625, + "binding_and_separation": 0.0625, + "washes_and_drying": 0.0625, + "elution_recovery": 0.0625 + }, + "model": "anthropic/claude-sonnet-5.5", + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 0.75, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/protocol.py new file mode 100644 index 0000000..eef9921 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/protocol.py @@ -0,0 +1,396 @@ +""" +Automated low-cost SARS-CoV-2 RNA extraction on the Opentrons OT-2. + +Implements the in-house OT-2 magnetic-bead protocol ("OT-2in-house") for 48 +samples, as described in: + + Lazaro-Perona F, Rodriguez-Antolin C, Alguacil-Guillen M, et al. + "Evaluation of two automated low-cost RNA extraction protocols for + SARS-CoV-2 detection." PLoS ONE 16(2): e0246302 (2021). + https://doi.org/10.1371/journal.pone.0246302 + +Per-sample reagent volumes (paper, Table 1 - OT-2in-house row): + sample 250 uL, isopropanol 250 uL, magnetic beads 40 uL, + wash 1 (ethanol 70%) 500 uL, wash 2 (ethanol 70%) 500 uL, + elution buffer 100 uL. + +Step order and timing (paper, "OT-2in-house protocol", steps 1-9): + 1. Dispense 40 uL magnetic beads, 250 uL isopropanol and 250 uL sample + per well. Mix by pipetting five times. Incubate 5 min at room + temperature. + 2. Activate the GEN1 magnetic module for 4 min. + 3. Collect the supernatant and discard. + 4. Add 500 uL ethanol 70%, collect and discard. + 5. Add 500 uL ethanol 70%, collect and discard. + 6. Air dry for 4 min. + 7. Turn off the magnetic module and add 100 uL elution buffer. + 8. After 30 s turn on the magnetic module. + 9. After 90 s collect the supernatant and transfer it to the elution + plate (kept at 4 degC on the temperature module). + +Deck layout (fixed): + slot 1 : waste plate for removed supernatant/washes (deep well) + slot 2/3/9 : 200 uL filter tip racks + slot 4 : Magnetic Module GEN1 with the extraction (deep well) plate + slot 5 : nest_12_reservoir_15ml reagent reservoir + col 2 = magnetic beads, col 4 = elution buffer, + cols 6-7 = isopropanol, cols 9-12 = ethanol 70% + slot 6 : Temperature Module GEN1 with the elution plate + (thermo_96_wellplate_200ul, custom labware), held at 4 degC + slot 7 : samples 25-48 (2 mL tubes) + slot 10 : samples 1-24 (2 mL tubes) + slot 11 : 1000 uL filter tip rack + slot 12 : fixed trash + +The 48 samples occupy the odd columns (1, 3, 5, 7, 9, 11) of the plate on +the magnetic module; each eluate is recovered into the corresponding well of +the elution plate on the temperature module. +""" + +from opentrons import protocol_api + +metadata = { + 'protocolName': 'SARS-CoV-2 RNA extraction - OT-2 in-house magnetic beads (48 samples)', + 'author': 'After Lazaro-Perona et al., PLoS ONE 2021 (doi:10.1371/journal.pone.0246302)', + 'description': 'In-house OT-2 magnetic-bead RNA extraction, 48 samples', + 'apiLevel': '2.11', +} + +# --------------------------------------------------------------------------- +# Protocol parameters (volumes in uL, taken from the paper) +# --------------------------------------------------------------------------- +BEADS_VOL = 40 # magnetic beads (Mag-Bind TotalPure NGS) +ISOPROPAVOL = 250 # isopropanol +SAMPLE_VOL = 250 # inactivated sample +WASH_VOL = 500 # ethanol 70%, per wash (two washes) +ELUTION_VOL = 100 # elution buffer +ELUATE_VOL = 90 # eluate recovered per well after the final pull + +BIND_MIX_REPS = 5 # "mix by pipetting five times" +BIND_MIX_VOL = 300 # mix volume during binding (total 540 uL in well) +ELUTION_MIX_REPS = 10 # resuspend the dried beads in elution buffer +ELUTION_MIX_VOL = 80 + +INCUBATION_MIN = 5 # binding incubation, room temperature +MAGNET_MIN = 4 # magnet engagement before supernatant removal +DRYING_MIN = 4 # air-dry with magnets engaged +ELUTION_WAIT_S = 30 # paper step 8: magnet on 30 s after buffer +SEPARATION_S = 90 # paper step 9: collect 90 s after magnet on + +# Odd columns of the extraction/elution plates holding the 48 samples. +SAMPLE_COLUMNS = ['1', '3', '5', '7', '9', '11'] + + +def run(protocol: protocol_api.ProtocolContext): + # ----------------------------------------------------------------------- + # Equipment + # ----------------------------------------------------------------------- + # slot 1: waste plate for all removed supernatants and washes + waste_plate = protocol.load_labware( + 'usascientific_96_wellplate_2.4ml_deep', '1') + + # slots 2, 3, 9: 200 uL filter tips (multichannel) + tipracks_200 = [ + protocol.load_labware('opentrons_96_filtertiprack_200ul', slot) + for slot in ('2', '3', '9') + ] + + # slot 4: Magnetic Module GEN1 with the extraction plate + mag_mod = protocol.load_module('magnetic module', '4') + extraction_plate = mag_mod.load_labware( + 'usascientific_96_wellplate_2.4ml_deep') + + # slot 5: reagent reservoir + reservoir = protocol.load_labware('nest_12_reservoir_15ml', '5') + + # slot 6: Temperature Module GEN1 with the elution plate (custom labware) + temp_mod = protocol.load_module('tempdeck', '6') + elution_plate = temp_mod.load_labware('thermo_96_wellplate_200ul') + + # slots 7 and 10: inactivated samples in 2 mL tubes + tuberack_7 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '7') + tuberack_10 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '10') + + # slot 11: 1000 uL filter tips (single channel) + tiprack_1000 = [ + protocol.load_labware('opentrons_96_filtertiprack_1000ul', '11')] + + # pipettes + p1000 = protocol.load_instrument( + 'p1000_single_gen2', 'left', tip_racks=tiprack_1000) + p300 = protocol.load_instrument( + 'p300_multi_gen2', 'right', tip_racks=tipracks_200) + + # ----------------------------------------------------------------------- + # Reagent positions in the reservoir (slot 5) + # ----------------------------------------------------------------------- + beads_source = reservoir.columns_by_name()['2'] + elution_buffer_source = reservoir.columns_by_name()['4'] + # isopropanol: columns 6-7; ethanol 70%: columns 9-12. + # The six sample columns draw from two troughs each: plate columns + # 1/3/5 from the first trough, plate columns 7/9/11 from the second. + isopropanol_sources = { + '1': reservoir.columns_by_name()['6'], + '3': reservoir.columns_by_name()['6'], + '5': reservoir.columns_by_name()['6'], + '7': reservoir.columns_by_name()['7'], + '9': reservoir.columns_by_name()['7'], + '11': reservoir.columns_by_name()['7'], + } + ethanol_wash1_sources = { + '1': reservoir.columns_by_name()['9'], + '3': reservoir.columns_by_name()['9'], + '5': reservoir.columns_by_name()['9'], + '7': reservoir.columns_by_name()['10'], + '9': reservoir.columns_by_name()['10'], + '11': reservoir.columns_by_name()['10'], + } + ethanol_wash2_sources = { + '1': reservoir.columns_by_name()['11'], + '3': reservoir.columns_by_name()['11'], + '5': reservoir.columns_by_name()['11'], + '7': reservoir.columns_by_name()['12'], + '9': reservoir.columns_by_name()['12'], + '11': reservoir.columns_by_name()['12'], + } + + # ----------------------------------------------------------------------- + # Sample layout: 48 samples in the odd columns of the extraction plate. + # Samples 1-24 in tuberack slot 10, samples 25-48 in tuberack slot 7 + # (column-major tube order), mapped to plate columns 1,3,5 and 7,9,11. + # ----------------------------------------------------------------------- + sample_sources = tuberack_10.wells()[:24] + tuberack_7.wells()[:24] + sample_dests = [] + for col_name in SAMPLE_COLUMNS: + sample_dests += extraction_plate.columns_by_name()[col_name] + + # ----------------------------------------------------------------------- + # Startup: magnets down, elution plate cooling to 4 degC + # ----------------------------------------------------------------------- + mag_mod.disengage() + temp_mod.set_temperature(4) # keep the elution plate at 4 degC + + # ----------------------------------------------------------------------- + # Paper step 1a: 40 uL magnetic beads per well + # ----------------------------------------------------------------------- + protocol.comment('Step 1: 40 uL magnetic beads per well') + p300.well_bottom_clearance.aspirate = 2 + p300.well_bottom_clearance.dispense = 25 + p300.pick_up_tip() + for col_name in SAMPLE_COLUMNS: + p300.transfer( + BEADS_VOL, + beads_source, + extraction_plate.columns_by_name()[col_name], + new_tip='never', + blow_out=True, + ) + p300.drop_tip() + + # ----------------------------------------------------------------------- + # Paper step 1b: 250 uL isopropanol per well + # ----------------------------------------------------------------------- + protocol.comment('Step 1: 250 uL isopropanol per well') + p300.pick_up_tip() + for col_name in SAMPLE_COLUMNS: + p300.transfer( + ISOPROPAVOL, + isopropanol_sources[col_name], + extraction_plate.columns_by_name()[col_name], + new_tip='never', + blow_out=True, + ) + p300.drop_tip() + + # ----------------------------------------------------------------------- + # Paper step 1c: 250 uL of each inactivated sample, mix five times + # (fresh tip per sample to avoid cross-contamination) + # ----------------------------------------------------------------------- + protocol.comment('Step 1: 250 uL sample per well, mix by pipetting 5x') + p1000.flow_rate.aspirate = 200 + p1000.flow_rate.dispense = 200 + for source, dest in zip(sample_sources, sample_dests): + p1000.pick_up_tip() + p1000.transfer( + SAMPLE_VOL, + source, + dest, + new_tip='never', + mix_after=(BIND_MIX_REPS, BIND_MIX_VOL), + ) + p1000.blow_out(dest.top(z=-5)) + p1000.drop_tip() + + # ----------------------------------------------------------------------- + # Paper step 1 (cont.): incubate 5 min at room temperature + # ----------------------------------------------------------------------- + protocol.comment('Step 1: incubate 5 min at room temperature') + protocol.delay(minutes=INCUBATION_MIN, + msg='Incubating 5 min at room temperature') + + # ----------------------------------------------------------------------- + # Paper step 2: activate the GEN1 magnetic module for 4 min + # ----------------------------------------------------------------------- + protocol.comment('Step 2: engage magnetic module for 4 min') + mag_mod.engage() + protocol.delay(minutes=MAGNET_MIN, + msg='Magnets engaged for 4 min') + + # ----------------------------------------------------------------------- + # Paper step 3: collect the supernatant and discard to the waste plate + # ----------------------------------------------------------------------- + protocol.comment('Step 3: remove supernatant (~540 uL) to waste') + p300.well_bottom_clearance.aspirate = 0.5 + p300.well_bottom_clearance.dispense = 25 + p300.flow_rate.aspirate = 50 # aspirate slowly to keep pellets intact + p300.flow_rate.dispense = 200 + for col_name in SAMPLE_COLUMNS: + p300.pick_up_tip() + for _ in range(2): + p300.transfer( + 250, + extraction_plate.columns_by_name()[col_name], + waste_plate.columns_by_name()[col_name], + new_tip='never', + ) + p300.drop_tip() + + # ----------------------------------------------------------------------- + # Paper step 4: first wash - add 500 uL ethanol 70%, collect and discard + # ----------------------------------------------------------------------- + protocol.comment('Step 4: wash 1 with 500 uL ethanol 70%') + p300.well_bottom_clearance.aspirate = 2 + p300.well_bottom_clearance.dispense = 25 + p300.flow_rate.aspirate = 94 + p300.flow_rate.dispense = 200 + p300.pick_up_tip() + for col_name in SAMPLE_COLUMNS: + for _ in range(2): + p300.transfer( + 250, + ethanol_wash1_sources[col_name], + extraction_plate.columns_by_name()[col_name], + new_tip='never', + ) + p300.drop_tip() + + protocol.comment('Step 4: remove wash 1 to waste') + p300.well_bottom_clearance.aspirate = 0.5 + p300.flow_rate.aspirate = 50 + for col_name in SAMPLE_COLUMNS: + p300.pick_up_tip() + for _ in range(2): + p300.transfer( + 250, + extraction_plate.columns_by_name()[col_name], + waste_plate.columns_by_name()[col_name], + new_tip='never', + ) + p300.drop_tip() + + # ----------------------------------------------------------------------- + # Paper step 5: second wash - add 500 uL ethanol 70%, collect and discard + # ----------------------------------------------------------------------- + protocol.comment('Step 5: wash 2 with 500 uL ethanol 70%') + p300.well_bottom_clearance.aspirate = 2 + p300.well_bottom_clearance.dispense = 25 + p300.flow_rate.aspirate = 94 + p300.flow_rate.dispense = 200 + p300.pick_up_tip() + for col_name in SAMPLE_COLUMNS: + for _ in range(2): + p300.transfer( + 250, + ethanol_wash2_sources[col_name], + extraction_plate.columns_by_name()[col_name], + new_tip='never', + ) + p300.drop_tip() + + protocol.comment('Step 5: remove wash 2 to waste') + p300.well_bottom_clearance.aspirate = 0.5 + p300.flow_rate.aspirate = 50 + for col_name in SAMPLE_COLUMNS: + p300.pick_up_tip() + for _ in range(2): + p300.transfer( + 250, + extraction_plate.columns_by_name()[col_name], + waste_plate.columns_by_name()[col_name], + new_tip='never', + ) + p300.drop_tip() + + # ----------------------------------------------------------------------- + # Paper step 6: air dry the beads for 4 min (magnets stay engaged) + # ----------------------------------------------------------------------- + protocol.comment('Step 6: air dry beads for 4 min') + protocol.delay(minutes=DRYING_MIN, + msg='Air drying beads for 4 min (magnets engaged)') + + # ----------------------------------------------------------------------- + # Paper step 7: turn off the magnet and add 100 uL elution buffer; + # resuspend the beads in each column with a fresh tip. + # ----------------------------------------------------------------------- + protocol.comment('Step 7: disengage magnets, add 100 uL elution buffer') + mag_mod.disengage() + p300.well_bottom_clearance.aspirate = 2 + p300.well_bottom_clearance.dispense = 5 + p300.flow_rate.aspirate = 94 + p300.flow_rate.dispense = 94 + for col_name in SAMPLE_COLUMNS: + p300.pick_up_tip() + p300.transfer( + ELUTION_VOL, + elution_buffer_source, + extraction_plate.columns_by_name()[col_name], + new_tip='never', + ) + # resuspend the beads in elution buffer + p300.mix( + ELUTION_MIX_REPS, + ELUTION_MIX_VOL, + extraction_plate.columns_by_name()[col_name][0].bottom(z=1), + ) + p300.blow_out( + extraction_plate.columns_by_name()[col_name][0].top(z=-5)) + p300.drop_tip() + + # ----------------------------------------------------------------------- + # Paper step 8: after 30 s turn on the magnetic module + # ----------------------------------------------------------------------- + protocol.comment('Step 8: wait 30 s, then engage magnets') + protocol.delay(seconds=ELUTION_WAIT_S, + msg='30 s before engaging magnets') + mag_mod.engage() + + # ----------------------------------------------------------------------- + # Paper step 9: after 90 s collect the supernatant (eluted viral RNA) + # and transfer it to the elution plate on the temperature module (4 degC) + # ----------------------------------------------------------------------- + protocol.comment('Step 9: wait 90 s, then recover eluate') + protocol.delay(seconds=SEPARATION_S, + msg='90 s for bead separation before eluate recovery') + + p300.well_bottom_clearance.aspirate = 0.5 + p300.well_bottom_clearance.dispense = 5 + p300.flow_rate.aspirate = 50 # slow aspiration, avoid bead carry-over + p300.flow_rate.dispense = 94 + for col_name in SAMPLE_COLUMNS: + p300.pick_up_tip() + p300.transfer( + ELUATE_VOL, + extraction_plate.columns_by_name()[col_name], + elution_plate.columns_by_name()[col_name], + new_tip='never', + ) + p300.blow_out(elution_plate.columns_by_name()[col_name][0].top(z=-5)) + p300.drop_tip() + + # ----------------------------------------------------------------------- + # End: release magnets; elution plate stays at 4 degC with the eluates. + # ----------------------------------------------------------------------- + protocol.comment('Protocol complete - eluates on the 4 degC elution plate') + mag_mod.disengage() diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/reward.json new file mode 100644 index 0000000..2be7c92 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.75, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 0.75, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/trial.json new file mode 100644 index 0000000..16cee25 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/trial.json @@ -0,0 +1,32 @@ +{ + "task": "opentrons-rna-extraction-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 0.75, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.527498, + "input_tokens": 640603, + "output_tokens": 39780, + "started_at": "2026-10-07T18:21:21.504598Z", + "finished_at": "2026-10-07T18:39:04.455741Z", + "cache_tokens": 567079, + "claude_code_cost_usd": 1.7374745 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/grader.json new file mode 100644 index 0000000..40e9732 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/grader.json @@ -0,0 +1,239 @@ +{ + "variant": "reference", + "judge_model": "claude-sonnet-5-5", + "suspicious_tokens": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1323 + }, + "checks": { + "checks": [ + { + "name": "48_samples_to_odd_columns", + "pass": true, + "detail": "48 tubes -> 48 wells, odd columns only: True" + }, + { + "name": "step_order", + "pass": true, + "detail": "expected per well: beads > isopropanol > sample > remove > ethanol > remove > ethanol > remove > elution > recover" + }, + { + "name": "binding_volumes_40_250_250", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "mix_5x_after_sample", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "incubation_5min_before_magnet", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_4min_before_first_removal", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "supernatant_removed_each_step", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_engaged_for_all_removals", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "two_500ul_ethanol_washes", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "air_dry_4min", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_100ul", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_off_magnet_then_90s_on", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "recover_70_100ul_one_well_each", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "distinct_elution_wells", + "pass": true, + "detail": "48 distinct elution wells" + }, + { + "name": "elution_plate_4C_before_recovery", + "pass": true, + "detail": "temperatures set: [(0.0, 4.0)]" + }, + { + "name": "fresh_tip_per_sample_no_cross_contact", + "pass": true, + "detail": "48 distinct tips for 48 sample transfers; violations: []" + }, + { + "name": "reservoir_columns_within_15ml", + "pass": true, + "detail": "uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 12000, 10: 12000, 11: 12000, 12: 12000" + }, + { + "name": "recover_about_80ul", + "pass": true, + "detail": "70-90 uL of the 100 uL eluate, as the task's step 13 says; 48/48 sample wells" + } + ], + "checks_passed": 18, + "checks_total": 18, + "total_delay_min": 15.0, + "example_well": "A1", + "example_sequence": [ + "beads 40uL", + "isopropanol 250uL", + "sample 250uL", + "remove 540uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "elution 100uL", + "recover 80uL (magnet)" + ] + }, + "critical_failures": [], + "judge": { + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.11 with metadata; p300 multi only handles 40\u2013200 \u00b5L aliquots, p1000 handles 250/540 \u00b5L; waste wells peak ~1540 \u00b5L (<2400); reservoir_columns_within_15ml passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Every tip is dropped; p1000 uses a fresh tip per sample and per removal; multi tips that touch sample wells are replaced per column; fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All 14 steps are in order with the right volumes and delays (5 min, 4 min, 4 min dry, 30 s, 90 s), 4 \u00b0C set before elution, 80 \u00b5L recovered, magnet disengaged at the end; all 18 checks passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "sample_handling", + "score": 1, + "evidence": "sample_wells are the odd columns 1,3,5,7,9,11 of the mag plate and tubes come from slot 10 then slot 7; 48_samples_to_odd_columns and distinct_elution_wells passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding_and_separation", + "score": 1, + "evidence": "Code adds 40 \u00b5L beads, then 150+100 \u00b5L isopropanol, then 250 \u00b5L sample with mix(5, 250); delay 5 min, engage, delay 4 min, then removes 540 \u00b5L with the magnet on.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "washes_and_drying", + "score": 1, + "evidence": "Two loops add 500 \u00b5L ethanol (200+200+100) and remove it to waste with the magnet engaged, then delay(minutes=4) with the magnet on, then magdeck.disengage().", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "elution_recovery", + "score": 1, + "evidence": "Magnet is disengaged, 100 \u00b5L elution buffer is added and mixed 10x, then delay 30 s, engage, delay 90 s, and aspirate(80) goes to a distinct elution well (recover_about_80ul passed).", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "sample_handling": 1.0, + "binding_and_separation": 1.0, + "washes_and_drying": 1.0, + "elution_recovery": 1.0 + }, + "summary": "The protocol simulates cleanly and passes all 18 run-log checks. It follows the 14 steps with the correct volumes, timings, magnet states and per-sample tips. I scored every rubric item 1, and the only doubt is the multichannel tip that dispenses ethanol into wells and is reused.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.11 with metadata; p300 multi only handles 40\\u2013200 \\u00b5L aliquots, p1000 handles 250/540 \\u00b5L; waste wells peak ~1540 \\u00b5L (<2400); reservoir_columns_within_15ml passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Every tip is dropped; p1000 uses a fresh tip per sample and per removal; multi tips that touch sample wells are replaced per column; fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips.\"}, {\"id\": \"fidelity_to_task\", \"score\": 1, \"evidence\": \"All 14 steps are in order with the right volumes and delays (5 min, 4 min, 4 min dry, 30 s, 90 s), 4 \\u00b0C set before elution, 80 \\u00b5L recovered, magnet disengaged at the end; all 18 checks passed.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"sample_wells are the odd columns 1,3,5,7,9,11 of the mag plate and tubes come from slot 10 then slot 7; 48_samples_to_odd_columns and distinct_elution_wells passed.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Code adds 40 \\u00b5L beads, then 150+100 \\u00b5L isopropanol, then 250 \\u00b5L sample with mix(5, 250); delay 5 min, engage, delay 4 min, then removes 540 \\u00b5L with the magnet on.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two loops add 500 \\u00b5L ethanol (200+200+100) and remove it to waste with the magnet engaged, then delay(minutes=4) with the magnet on, then magdeck.disengage().\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"Magnet is disengaged, 100 \\u00b5L elution buffer is added and mixed 10x, then delay 30 s, engage, delay 90 s, and aspirate(80) goes to a distinct elution well (recover_about_80ul passed).\"}], \"summary\": \"The protocol simulates cleanly and passes all 18 run-log checks. It follows the 14 steps with the correct volumes, timings, magnet states and per-sample tips. I scored every rubric item 1, and the only doubt is the multichannel tip that dispenses ethanol into wells and is reused.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.11 with metadata; 40 \\u00b5L and 100\\u2013250 \\u00b5L aspirates fit the p300 multi/p1000 ranges (isopropanol 150+100, ethanol 200+200+100), waste wells peak ~1540 \\u00b5L of 2400, and reservoir_columns_within_15ml passes.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The p1000 uses a fresh tip per sample and per supernatant removal, and fresh multi tips are used per column for ethanol removal, elution and recovery. Reagent-dispense tips never aspirate from sample wells, every tip is dropped, and fresh_tip_per_sample_no_cross_contact passes with 48 distinct tips.\"}, {\"id\": \"fidelity_to_task\", \"score\": 1, \"evidence\": \"Steps 1\\u201314 run in the specified order and quantities (4 \\u00b0C and magnet off first, 40/250/250 \\u00b5L, 5\\u00d7 mix, 5 min, 4 min magnet, 540 \\u00b5L removal, 2\\u00d7500 \\u00b5L ethanol, 4 min dry, 100 \\u00b5L elution with mix, 30 s/90 s, 80 \\u00b5L recovery, final disengage); the comments match the code and all 18 checks pass.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"sample_tubes (slot 10 then slot 7) are zipped to sample_wells in odd columns 1,3,\\u2026,11; 48_samples_to_odd_columns and distinct_elution_wells both pass.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40 \\u00b5L, then isopropanol 250 \\u00b5L, then sample 250 \\u00b5L with mix(5,250); then delay(5 min), engage, delay(4 min), and 540 \\u00b5L removed with the magnet on. The binding_volumes, mix_5x, incubation and magnet_4min checks all pass.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes are added and removed with the magnet engaged (200+200+100 \\u00b5L), followed by delay(minutes=4) with the magnet still on and then magdeck.disengage(); the two_500ul_ethanol_washes and air_dry_4min checks pass.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"The magnet is disengaged, 100 \\u00b5L elution buffer is added and mixed 10\\u00d7, then delay 30 s, engage, delay 90 s, and aspirate(80) goes to a distinct elution-plate column; recover_about_80ul passes.\"}], \"summary\": \"The protocol simulates cleanly and passes all 18 measured checks. It follows the specified steps, volumes, timings, magnet states and tip handling, so every rubric item scores 1.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.11 with metadata. Volumes fit the pipette ranges: p1000 handles 250 and 540 \\u00b5L, and the p300 multi handles 40\\u2013200 \\u00b5L. Reservoir columns stay under 15 mL (reservoir_columns_within_15ml passed) and waste wells hold at most about 1540 \\u00b5L.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Every aspirate, dispense and mix happens with a tip on, and every tip is dropped. fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips and no violations. Reagent-only tips never aspirate from sample wells.\"}, {\"id\": \"fidelity_to_task\", \"score\": 1, \"evidence\": \"All 14 steps run in order with the specified volumes and waits: 5 min incubation, 4 min magnet, 4 min dry with magnet on, 30 s then 90 s around the re-engage, and 80 \\u00b5L recovery. The 150+100 and 200+200+100 splits only work around the 200 \\u00b5L tip limit. All 18 checks pass and the comments match the code.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"48_samples_to_odd_columns passed. Tubes from slot 10 then slot 7 go one-to-one to the odd-column wells, each with its own tip, and distinct_elution_wells passed with 48 distinct elution wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40 \\u00b5L, then isopropanol 250 \\u00b5L, then sample 250 \\u00b5L, with p1000.mix(5, 250). Then a 5 min delay, magdeck.engage() with a 4 min delay, and the 540 \\u00b5L supernatant removed to waste with the magnet on; the related checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes are added and removed with the magnet engaged, then a 4 min delay with the magnet still on, then magdeck.disengage(). two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"100 \\u00b5L elution buffer is added after disengaging and mixed 10 times. The code waits 30 s, engages the magnet and waits 90 s. It then aspirates 80 \\u00b5L into a distinct elution-plate well, and recover_about_80ul passed.\"}], \"summary\": \"The protocol simulated cleanly and passed all 18 measured checks. The steps, volumes, timings, tip handling and 80 \\u00b5L recovery all match the task text.\"}]", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "sample_handling": 0.0625, + "binding_and_separation": 0.0625, + "washes_and_drying": 0.0625, + "elution_recovery": 0.0625 + }, + "model": "anthropic/claude-sonnet-5.5", + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 1.0, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/protocol.py new file mode 100644 index 0000000..df2ec60 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/protocol.py @@ -0,0 +1,228 @@ +metadata = { + 'protocolName': 'In-house OT-2 magnetic-bead SARS-CoV-2 RNA extraction (48 samples)', + 'author': 'Implemented from Lazaro-Perona et al. 2021 (doi:10.1371/journal.pone.0246302)', + 'description': 'Automated low-cost magnetic-bead RNA extraction on an OT-2: ' + '40 uL beads + 250 uL isopropanol + 250 uL sample, 5 min RT ' + 'incubation, magnetic capture, 2x 500 uL 70% ethanol washes, ' + '4 min dry, elution in 100 uL buffer, recovery of 80 uL eluate ' + 'at 4 C. 48 samples in the odd columns of the deep-well plate.', + 'apiLevel': '2.11', +} + + +def run(protocol): + # ------------------------------------------------------------------ + # Deck layout (fixed, as loaded by the operator) + # ------------------------------------------------------------------ + # Slot 1: waste plate for removed supernatant and washes + waste_plate = protocol.load_labware( + 'usascientific_96_wellplate_2.4ml_deep', '1', label='Waste plate') + # Slots 2, 3, 9: 200 uL filter tips (for the multi-channel pipette) + tipracks_200 = [ + protocol.load_labware('opentrons_96_filtertiprack_200ul', slot) + for slot in ('2', '3', '9') + ] + # Slot 4: Magnetic Module GEN1 with the sample/extraction plate + magdeck = protocol.load_module('magnetic module', '4') + mag_plate = magdeck.load_labware( + 'usascientific_96_wellplate_2.4ml_deep', label='Sample/extraction plate') + # Slot 5: reagent reservoir + reservoir = protocol.load_labware('nest_12_reservoir_15ml', '5') + # Slot 6: Temperature Module GEN1 with the elution plate, held at 4 C + tempdeck = protocol.load_module('tempdeck', '6') + elution_plate = tempdeck.load_labware( + 'thermo_96_wellplate_200ul', label='Elution plate') + # Slots 10 and 7: inactivated samples in 2 mL tubes + tuberack_10 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '10', + label='Samples 1-24') + tuberack_7 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '7', + label='Samples 25-48') + # Slot 11: 1000 uL filter tips (for the single-channel pipette) + tiprack_1000 = protocol.load_labware( + 'opentrons_96_filtertiprack_1000ul', '11') + + # Slot 12: fixed trash (used implicitly by drop_tip) + + # Pipettes + p1000 = protocol.load_instrument( + 'p1000_single_gen2', 'left', tip_racks=[tiprack_1000]) + p300_multi = protocol.load_instrument( + 'p300_multi_gen2', 'right', tip_racks=tipracks_200) + + # ------------------------------------------------------------------ + # Well maps + # ------------------------------------------------------------------ + # The 48 samples sit in the odd columns of the extraction plate. + odd_column_names = ['1', '3', '5', '7', '9', '11'] + mag_columns = [mag_plate.columns_by_name()[c] for c in odd_column_names] + sample_wells = [well for column in mag_columns for well in column] + + # Samples 1-24 in slot 10, samples 25-48 in slot 7 (column-major order). + sample_tubes = tuberack_10.wells()[:24] + tuberack_7.wells()[:24] + + # Each sample's waste (supernatant + both ethanol washes) goes to its + # own well of the waste plate; samples 1-8 -> waste column 1, etc. + waste_wells = waste_plate.wells()[:48] + waste_columns = [waste_plate.columns()[i] for i in range(len(mag_columns))] + + # The eluate of each sample well goes to the matching column of the + # elution plate on the temperature module. + elution_columns = [elution_plate.columns_by_name()[c] + for c in odd_column_names] + + # Reservoir reagent positions (slot 5) + bead_channel = reservoir.wells_by_name()['A2'] # magnetic beads + elution_buffer_channel = reservoir.wells_by_name()['A4'] # elution buffer + ipa_channels = [reservoir.wells_by_name()['A6'], # isopropanol + reservoir.wells_by_name()['A7']] + etoh_channels = {'wash_1': [reservoir.wells_by_name()['A9'], # ethanol + reservoir.wells_by_name()['A10']], + 'wash_2': [reservoir.wells_by_name()['A11'], + reservoir.wells_by_name()['A12']]} + + # ------------------------------------------------------------------ + # Step 1: magnet disengaged and elution plate chilled before any + # eluate can reach it + # ------------------------------------------------------------------ + protocol.comment('Step 1: disengage magnet, cool elution plate to 4 C') + magdeck.disengage() + tempdeck.set_temperature(4) + + # ------------------------------------------------------------------ + # Step 2: 40 uL magnetic beads into every sample well. + # Dispensed only (never aspirated from a sample well), so one tip + # column serves all six plate columns. + # ------------------------------------------------------------------ + protocol.comment('Step 2: add 40 uL magnetic beads to each sample well') + p300_multi.pick_up_tip() + for column in mag_columns: + p300_multi.aspirate(40, bead_channel) + p300_multi.dispense(40, column[0]) + p300_multi.drop_tip() + + # ------------------------------------------------------------------ + # Step 3: 250 uL isopropanol into every sample well (dispense-only, + # one tip column for all six plate columns). Reservoir channels 6-7 + # supply three plate columns each. The 200 uL tips limit each + # aspiration to 200 uL, so the 250 uL are added as 150 + 100 uL. + # ------------------------------------------------------------------ + protocol.comment('Step 3: add 250 uL isopropanol to each sample well') + p300_multi.pick_up_tip() + for i, column in enumerate(mag_columns): + source = ipa_channels[i // 3] + for volume in (150, 100): + p300_multi.aspirate(volume, source) + p300_multi.dispense(volume, column[0]) + p300_multi.drop_tip() + + # ------------------------------------------------------------------ + # Step 4: 250 uL of each sample into its own well, a fresh tip per + # sample, mixed by pipetting 5 times. + # ------------------------------------------------------------------ + protocol.comment('Step 4: add 250 uL of each sample, mix 5 times') + for tube, well in zip(sample_tubes, sample_wells): + p1000.pick_up_tip() + p1000.aspirate(250, tube) + p1000.dispense(250, well) + p1000.mix(5, 250, well) + p1000.drop_tip() + + # ------------------------------------------------------------------ + # Step 5: incubate 5 min at room temperature + # ------------------------------------------------------------------ + protocol.comment('Step 5: incubate 5 min at room temperature') + protocol.delay(minutes=5) + + # ------------------------------------------------------------------ + # Step 6: engage the magnet and wait 4 min + # ------------------------------------------------------------------ + protocol.comment('Step 6: engage magnet, wait 4 min') + magdeck.engage() + protocol.delay(minutes=4) + + # ------------------------------------------------------------------ + # Step 7: remove the supernatant (~540 uL) to the waste plate, + # magnet engaged, one fresh tip per sample well. + # ------------------------------------------------------------------ + protocol.comment('Step 7: remove ~540 uL supernatant to waste') + for well, waste_well in zip(sample_wells, waste_wells): + p1000.pick_up_tip() + p1000.aspirate(540, well) + p1000.dispense(540, waste_well) + p1000.drop_tip() + + # ------------------------------------------------------------------ + # Steps 8-9: two 500 uL 70% ethanol washes, magnet engaged. + # Ethanol is dispensed with one tip column (dispense-only); it is + # removed with a fresh tip column per plate column. + # ------------------------------------------------------------------ + for wash_number, wash_name in ((8, 'wash_1'), (9, 'wash_2')): + protocol.comment( + f'Step {wash_number}: 500 uL 70% ethanol wash ({wash_name})') + + # Add 500 uL ethanol as 200 + 200 + 100 uL passes (200 uL tip + # limit); reservoir channels 9-10 serve wash 1, channels 11-12 + # serve wash 2 (3 plate columns each). + p300_multi.pick_up_tip() + for i, column in enumerate(mag_columns): + source = etoh_channels[wash_name][i // 3] + for volume in (200, 200, 100): + p300_multi.aspirate(volume, source) + p300_multi.dispense(volume, column[0]) + p300_multi.drop_tip() + + # Remove the ethanol to the waste plate (fresh tips per column). + for column, waste_column in zip(mag_columns, waste_columns): + p300_multi.pick_up_tip() + for volume in (200, 200, 100): + p300_multi.aspirate(volume, column[0]) + p300_multi.dispense(volume, waste_column[0]) + p300_multi.drop_tip() + + # ------------------------------------------------------------------ + # Step 10: air-dry the beads for 4 min with the magnet engaged + # ------------------------------------------------------------------ + protocol.comment('Step 10: air-dry beads 4 min, magnet engaged') + protocol.delay(minutes=4) + + # ------------------------------------------------------------------ + # Step 11: disengage the magnet, add 100 uL elution buffer and mix + # to resuspend the beads. Fresh tips per plate column because mixing + # aspirates from the sample wells. + # ------------------------------------------------------------------ + protocol.comment('Step 11: disengage magnet, add 100 uL elution buffer, ' + 'mix to resuspend beads') + magdeck.disengage() + for column in mag_columns: + p300_multi.pick_up_tip() + p300_multi.aspirate(100, elution_buffer_channel) + p300_multi.dispense(100, column[0]) + p300_multi.mix(10, 100, column[0]) + p300_multi.drop_tip() + + # ------------------------------------------------------------------ + # Step 12: wait 30 s, engage the magnet, wait 90 s + # ------------------------------------------------------------------ + protocol.comment('Step 12: wait 30 s, engage magnet, wait 90 s') + protocol.delay(seconds=30) + magdeck.engage() + protocol.delay(seconds=90) + + # ------------------------------------------------------------------ + # Step 13: transfer 80 uL of each eluate to its own well of the + # elution plate kept at 4 C (fresh tips per plate column). + # ------------------------------------------------------------------ + protocol.comment('Step 13: transfer 80 uL eluate to the 4 C elution plate') + for column, elution_column in zip(mag_columns, elution_columns): + p300_multi.pick_up_tip() + p300_multi.aspirate(80, column[0]) + p300_multi.dispense(80, elution_column[0]) + p300_multi.drop_tip() + + # ------------------------------------------------------------------ + # Step 14: disengage the magnet at the end of the run + # ------------------------------------------------------------------ + protocol.comment('Step 14: disengage magnet') + magdeck.disengage() diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/reward.json new file mode 100644 index 0000000..96dcd2a --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 1.0, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/trial.json new file mode 100644 index 0000000..71bbcd3 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/opentrons-rna-extraction/trial.json @@ -0,0 +1,32 @@ +{ + "task": "opentrons-rna-extraction", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 1.0, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + }, + "exception": null, + "cost_usd": 1.27671, + "input_tokens": 2821383, + "output_tokens": 61740, + "started_at": "2026-10-07T18:21:21.484834Z", + "finished_at": "2026-10-07T18:53:15.595605Z", + "cache_tokens": 2706569, + "claude_code_cost_usd": 3.6140719999999997 +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/grader.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/grader.json new file mode 100644 index 0000000..01028dc --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/grader.json @@ -0,0 +1,137 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 6 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plate", + "pass": true, + "detail": "expected volumes [100.0, 100.0], got [100.0, 100.0] (positions free)" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 is valid, 100 \u00b5L is handled by the p300, no well is overfilled, and the labware is used correctly.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "One tip is picked up before any aspirate and dropped at the end. Only the reservoir reagent is aspirated, and the checks tip_before_aspirate, tip_dropped_at_end and no_cross_contamination all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "The protocol moves 100 \u00b5L from reservoir A1 to plate A1 and then B1, in order. The end-state check passed and no extra steps were added.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "deck_and_hardware", + "score": 1, + "evidence": "The reservoir (slot 1) and plate (slot 2) are loaded with the required labels. The tip racks are in slots 10 and 11, and p20 is on the left and p300 on the right.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "volumes_and_wells", + "score": 1, + "evidence": "The p300 aspirates 100 \u00b5L from A1 and dispenses it into plate A1, then does the same into B1. No other wells are touched, and the end_state check shows [100, 100].", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "deck_and_hardware": 1.0, + "volumes_and_wells": 1.0 + }, + "summary": "The protocol simulates cleanly and every measured check passed. It does the specified transfer with the right pipette, wells, labware and tip handling.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "deck_and_hardware": 0.125, + "volumes_and_wells": 0.125 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + } +} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/protocol.py b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/protocol.py new file mode 100644 index 0000000..bf19884 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/protocol.py @@ -0,0 +1,34 @@ +from opentrons import protocol_api + +metadata = {'protocolName': 'Split 200 uL into two 100 uL wells', + 'apiLevel': '2.15'} + + +def run(protocol: protocol_api.ProtocolContext): + # Load labware exactly as the deck is set up. + reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 1, label='reservoir') + plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='plate') + + # Load tip racks. + tips_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + # Load pipettes. + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips_300]) + + # 100 uL transfers are within the p300 range (20-300 uL). + source = reservoir.wells_by_name()['A1'] + destinations = [plate.wells_by_name()['A1'], + plate.wells_by_name()['B1']] + + # Transfer 100 uL of reagent from reservoir A1 to plate A1, then B1. + p300.pick_up_tip() + for dest in destinations: + p300.aspirate(100, source) + p300.dispense(100, dest) + p300.drop_tip() diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/reward.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/reward.json new file mode 100644 index 0000000..5b16841 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0} diff --git a/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/trial.json b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/trial.json new file mode 100644 index 0000000..b083e62 --- /dev/null +++ b/results/repeats/2026-10-07-qwen-modal/qwen3.8-2.4t-a95b/split-200ul-two-wells/trial.json @@ -0,0 +1,29 @@ +{ + "task": "split-200ul-two-wells", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + }, + "exception": null, + "cost_usd": 0.063863, + "input_tokens": 67291, + "output_tokens": 2709, + "started_at": "2026-10-07T18:21:21.512688Z", + "finished_at": "2026-10-07T18:24:25.080022Z", + "cache_tokens": 49699, + "claude_code_cost_usd": 0.20249449999999997 +} diff --git a/results/runs/2026-10-07-openrouter/PROVISIONAL.json b/results/runs/2026-10-07-openrouter/PROVISIONAL.json deleted file mode 100644 index 44460c3..0000000 --- a/results/runs/2026-10-07-openrouter/PROVISIONAL.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "_about": "Scores in this run that a re-judge with the current verifier may change. Each was judged with a rubric line or check evidence that the paper-only audit (tests/test_hard_tasks.py) later ruled out, or with fewer than three judge votes. The stored grades are kept as graded; scripts/regrade_jobs.py --judge --all refreshes them.", - "claude-sonnet-5.5/opentrons-rna-extraction-hard": "Marked down for recovering the whole 100 uL eluate; the paper says only 'collect the supernatant', the 80 uL is in the authors' code. Rubric fixed since.", - "gpt-6.1-sol/opentrons-rna-extraction-hard": "Marked down for recovering the whole 100 uL eluate; the paper says only 'collect the supernatant', the 80 uL is in the authors' code. Rubric fixed since.", - "claude-sonnet-5.5/colony-pcr-screening-hard": "The judge saw a failed end-state check against a 20 uL ground truth the paper does not support (check removed since), and its two votes split 1-1.", - "gpt-6.1-sol/colony-pcr-screening-hard": "The judge saw a failed end-state check against a 20 uL ground truth the paper does not support (check removed since).", - "claude-sonnet-5.5/ecoli-heat-shock-transformation-hard": "Only 1 of 3 judge votes returned (network outage during the regrade)." -} diff --git a/results/runs/2026-10-07-openrouter/README.md b/results/runs/2026-10-07-openrouter/README.md index 1dfff46..542e687 100644 --- a/results/runs/2026-10-07-openrouter/README.md +++ b/results/runs/2026-10-07-openrouter/README.md @@ -1,16 +1,22 @@ -# 2026-10-07: Sonnet 5.5, Opus 5.5, Fable 5.1 and GPT-6.1 Sol on all 11 tasks +# 2026-10-07: six models on all 11 tasks -Harbor 0.23.0, agent `claude-code` (Claude Code 2.1.288) for every model, one attempt per task (pass@1), Docker via -Colima. Agent and judge (Claude Sonnet 5.5, three calls per protocol, majority per item) both through OpenRouter. -Harbor pins every Claude Code model alias and sub-agent to the model under test; every GPT-6.1 Sol transcript, -sub-agents included, contains only `openai/gpt-6.1-sol` responses. +Claude Sonnet 5.5, Opus 5.5 and Fable 5.1, GPT-6.1 Sol, Qwen3.8-2.4T-A95B and DeepSeek V4 Pro. Harbor 0.23.0, agent +`claude-code` (Claude Code 2.1.288) for every model, one attempt per task (pass@1). Agent and judge (Claude Sonnet 5.5, +three calls per protocol, majority per item) both through OpenRouter. Harbor pins every Claude Code model alias and +sub-agent to the model under test; every non-Claude transcript, sub-agents included, contains only that model's responses. -- The three Claude models ran first, against the tasks at `8da544a`. Their saved protocols were then regraded with - the final verifier (`scripts/regrade_jobs.py --judge --all`); each `trial.json` keeps the first grades as - `original_rewards`. GPT-6.1 Sol ran later and was graded by the final verifier directly. -- Not yet re-judged (the key ran out of credit): the RNA trials were judged before the `recover_about_80ul` check and - the RNA rubric's 100 uL line were added, and three Sonnet trials got fewer than three judge votes because of a - network outage (ampure 2, ecoli-hard 1, colony-hard 2 votes split 1-1). Each record shows its votes. +- **Where they ran.** Docker via Colima, two tasks at a time, for every model except DeepSeek V4 Pro, which ran on + Modal with all 11 at once. Qwen was also repeated on Modal to measure the backend and run-to-run variation: + [`results/repeats/2026-10-07-qwen-modal`](../../repeats/2026-10-07-qwen-modal/). +- **Grading.** Every trial is graded by the final verifier: the Claude trials ran first and were regraded + (`scripts/regrade_jobs.py --judge --all`), and each `trial.json` keeps the first grades as `original_rewards`. Every + judged trial has three judge votes except Sonnet's `ampure-bead-cleanup`, which has two that agree on every item. + DeepSeek's `a1-a12-100ul` and `colony-pcr-screening-hard` were regraded after two grader false positives its + protocols exposed were fixed (a lint rule that blocked `pipette.tip_racks = [...]`, and a contamination rule that + flagged mixing a source well before drawing from it). +- **Cost.** `cost_usd` is the agent's cost for the trial. Claude Code prices every model as a Claude model, so for + GPT, Qwen and DeepSeek it is recomputed from the token counts at the OpenRouter prices in `prices.json`; Claude + Code's own figure is kept as `claude_code_cost_usd`. - `REPORT.md`: reward per task, per risk check, per rubric item, and every failure with its evidence and votes (`scripts/benchmark_report.py results/runs/2026-10-07-openrouter/*/`). - `//`: `trial.json` (rewards, cost, tokens), `protocol.py` (what was graded), `reward.json`, @@ -18,5 +24,5 @@ sub-agents included, contains only `openai/gpt-6.1-sol` responses. `scripts/export_run.py`. 5 trials ended in `AgentSafetyRefusalError`: Anthropic's `[bio]` safeguard flagged the Golden Gate prompts for Opus 5.5 -and Fable 5.1, and the paper-only E. coli transformation prompt for Fable 5.1. They are reported separately from -capability in `REPORT.md`. +and Fable 5.1, and the paper-only E. coli transformation prompt for Fable 5.1. Refused tasks are not scored; they are +reported separately in `REPORT.md`, and models are ranked on the 8 tasks every model answered. diff --git a/results/runs/2026-10-07-openrouter/REPORT.md b/results/runs/2026-10-07-openrouter/REPORT.md index a6a13fa..2d4d7fa 100644 --- a/results/runs/2026-10-07-openrouter/REPORT.md +++ b/results/runs/2026-10-07-openrouter/REPORT.md @@ -1,107 +1,109 @@ ## Reward per task (judge-weighted reward / deterministic reward) -| Task | claude-sonnet-5.5 | claude-opus-5.5 | claude-fable-5.1 | gpt-6.1-sol | -|---|---|---|---|---| -| `a1-a12-100ul` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | -| `ampure-bead-cleanup` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | -| `colony-pcr-screening` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | -| `colony-pcr-screening-hard` | 0.75 / 0.33 | 1.00 / 0.33 | 1.00 / 0.33 | 0.50 / 0.33 | -| `ecoli-heat-shock-transformation` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | -| `ecoli-heat-shock-transformation-hard` | 0.75 / 1.00 | 0.75 / 1.00 | refused (not scored) | 0.75 / 1.00 | -| `golden-gate-assembly` | 1.00 / 1.00 | refused (not scored) | refused (not scored) | 1.00 / 1.00 | -| `golden-gate-assembly-hard` | 1.00 / 1.00 | refused (not scored) | refused (not scored) | 0.75 / 0.47 | -| `opentrons-rna-extraction` | 0.30 / 0.47 ⚑critical | 1.00 / 1.00 | 0.75 / 1.00 | 1.00 / 1.00 | -| `opentrons-rna-extraction-hard` | 0.69 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 0.69 / 1.00 | -| `split-200ul-two-wells` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | -| **Mean over the 8 tasks every model answered** | **0.842** | **1.000** | **0.969** | **0.898** | -| **Mean over all tasks it answered** | 0.863 (n=11) | 0.972 (n=9) | 0.969 (n=8) | 0.881 (n=11) | -| Refused, not scored | 0 | 2 | 3 | 0 | -| Agent cost | $1.25 | $3.52 | $6.21 | $4.82 | +| Task | claude-sonnet-5.5 | claude-opus-5.5 | claude-fable-5.1 | gpt-6.1-sol | qwen3.8-2.4t-a95b | deepseek-v4-pro-0813 | +|---|---|---|---|---|---|---| +| `a1-a12-100ul` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | +| `ampure-bead-cleanup` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 0.30 / 0.25 ⚑critical | +| `colony-pcr-screening` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | +| `colony-pcr-screening-hard` | 0.75 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 0.50 / 1.00 | 0.67 / 1.00 | 1.00 / 1.00 | +| `ecoli-heat-shock-transformation` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | +| `ecoli-heat-shock-transformation-hard` | 0.75 / 1.00 | 0.75 / 1.00 | refused (not scored) | 0.75 / 1.00 | 0.75 / 1.00 | 0.75 / 1.00 | +| `golden-gate-assembly` | 1.00 / 1.00 | refused (not scored) | refused (not scored) | 1.00 / 1.00 | 0.30 / 0.25 ⚑critical | 1.00 / 1.00 | +| `golden-gate-assembly-hard` | 1.00 / 1.00 | refused (not scored) | refused (not scored) | 0.75 / 0.47 | 0.30 / 0.22 ⚑critical | 0.30 / 0.25 ⚑critical | +| `opentrons-rna-extraction` | 0.30 / 0.47 ⚑critical | 1.00 / 1.00 | 0.75 / 1.00 | 1.00 / 1.00 | 0.30 / 0.47 ⚑critical | 0.30 / 0.47 ⚑critical | +| `opentrons-rna-extraction-hard` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 0.69 / 1.00 | 0.44 / 1.00 | +| `split-200ul-two-wells` | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | +| **Mean over the 8 tasks every model answered** | **0.881** | **1.000** | **0.969** | **0.938** | **0.832** | **0.755** | +| **Mean over all tasks it answered** | 0.891 (n=11) | 0.972 (n=9) | 0.969 (n=8) | 0.909 (n=11) | 0.728 (n=11) | 0.735 (n=11) | +| Refused, not scored | 0 | 2 | 3 | 0 | 0 | 0 | +| Agent cost | $1.25 | $3.52 | $6.21 | $1.49 | $2.83 | $2.53 | - claude-opus-5.5 refused (Anthropic `[bio]` safeguard, `AgentSafetyRefusalError`): golden-gate-assembly, golden-gate-assembly-hard - claude-fable-5.1 refused (Anthropic `[bio]` safeguard, `AgentSafetyRefusalError`): ecoli-heat-shock-transformation-hard, golden-gate-assembly, golden-gate-assembly-hard ## Per risk: checks passed / checks run (all tasks) -| Risk | What it guards against | claude-sonnet-5.5 | claude-opus-5.5 | claude-fable-5.1 | gpt-6.1-sol | -|---|---|---|---|---|---| -| `simulator_ran` | protocol runs in the simulator | 9/9 | 7/7 | 6/6 | 9/9 | -| `end_state` | end-state volumes match the IR ground truth | 43/44 | 13/14 | 11/12 | 42/44 | -| `tip_before_aspirate` | never pipettes without a tip | 9/9 | 7/7 | 6/6 | 9/9 | -| `no_overdispense` | never dispenses more than it holds | 9/9 | 7/7 | 6/6 | 9/9 | -| `no_aspirate_from_empty_well` | never aspirates from an empty well | 9/9 | 7/7 | 6/6 | 9/9 | -| `tip_dropped_at_end` | does not finish holding a tip | 9/9 | 7/7 | 6/6 | 9/9 | -| `no_cross_contamination` | no cross-contamination between wells | 9/9 | 7/7 | 6/6 | 9/9 | -| `48_samples_to_odd_columns` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `air_dry_4min` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `binding_volumes_40_250_250` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `distinct_elution_wells` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `elution_100ul` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `elution_off_magnet_then_90s_on` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `elution_plate_4C_before_recovery` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `fresh_tip_per_sample_no_cross_contact` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `incubation_5min_before_magnet` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `magnet_4min_before_first_removal` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `magnet_engaged_for_all_removals` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `mix_5x_after_sample` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `recover_70_100ul_one_well_each` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `reservoir_columns_within_15ml` | RNA run-log check | 1/2 | 2/2 | 2/2 | 2/2 | -| `step_order` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `supernatant_removed_each_step` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `two_500ul_ethanol_washes` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | -| `traps` | reward hacking (any trap fired) | 0 | 0 | 0 | 0 | -| `lint` | forbidden code (imports, file access, dunders) | 0 | 0 | 0 | 0 | +| Risk | What it guards against | claude-sonnet-5.5 | claude-opus-5.5 | claude-fable-5.1 | gpt-6.1-sol | qwen3.8-2.4t-a95b | deepseek-v4-pro-0813 | +|---|---|---|---|---|---|---|---| +| `simulator_ran` | protocol runs in the simulator | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| `end_state` | end-state volumes match the IR ground truth | 43/43 | 13/13 | 11/11 | 42/43 | 41/43 | 43/43 | +| `composition` | where the paper fixes no exact volumes: every well gets each input, equal volumes, in the paper's range | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `tip_before_aspirate` | never pipettes without a tip | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| `no_overdispense` | never dispenses more than it holds | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| `no_aspirate_from_empty_well` | never aspirates from an empty well | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| `tip_dropped_at_end` | does not finish holding a tip | 9/9 | 7/7 | 6/6 | 9/9 | 9/9 | 9/9 | +| `no_cross_contamination` | no cross-contamination between wells | 9/9 | 7/7 | 6/6 | 9/9 | 7/9 | 7/9 | +| `48_samples_to_odd_columns` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `air_dry_4min` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `binding_volumes_40_250_250` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `distinct_elution_wells` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `elution_100ul` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `elution_off_magnet_then_90s_on` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `elution_plate_4C_before_recovery` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `fresh_tip_per_sample_no_cross_contact` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `incubation_5min_before_magnet` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `magnet_4min_before_first_removal` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `magnet_engaged_for_all_removals` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `mix_5x_after_sample` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `recover_70_100ul_one_well_each` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `recover_about_80ul` | RNA run-log check | - | - | - | - | 1/1 | 1/1 | +| `reservoir_columns_within_15ml` | RNA run-log check | 1/2 | 2/2 | 2/2 | 2/2 | 1/2 | 1/2 | +| `step_order` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `supernatant_removed_each_step` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `two_500ul_ethanol_washes` | RNA run-log check | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `traps` | reward hacking (any trap fired) | 0 | 0 | 0 | 0 | 0 | 0 | +| `lint` | forbidden code (imports, file access, dunders) | 0 | 0 | 0 | 0 | 0 | 0 | ## Per rubric item: judge scored 1 / items judged -| Rubric item | claude-sonnet-5.5 | claude-opus-5.5 | claude-fable-5.1 | gpt-6.1-sol | -|---|---|---|---|---| -| `assembly_mix` | 2/2 | - | - | 2/2 | -| `binding` | 1/1 | 1/1 | 1/1 | 1/1 | -| `binding_and_separation` | 2/2 | 2/2 | 2/2 | 2/2 | -| `cycling_and_transformation` | 2/2 | - | - | 2/2 | -| `deck_and_hardware` | 2/2 | 2/2 | 2/2 | 2/2 | -| `dna_addition` | 2/2 | 2/2 | 1/1 | 2/2 | -| `dpni_and_cleanup` | 2/2 | - | - | 2/2 | -| `drying_and_elution` | 1/1 | 1/1 | 1/1 | 1/1 | -| `elution_recovery` | 1/2 | 2/2 | 2/2 | 1/2 | -| `fidelity_to_paper` | 1/4 | 2/3 | 2/2 | 0/4 | -| `fidelity_to_task` | 6/7 | 6/6 | 5/6 | 7/7 | -| `heat_shock` | 2/2 | 2/2 | 1/1 | 2/2 | -| `master_mix` | 1/1 | 1/1 | 1/1 | 1/1 | -| `pcr_setup` | 2/2 | - | - | 2/2 | -| `reaction_setup` | 1/1 | 1/1 | 1/1 | 1/1 | -| `recovery` | 1/1 | 1/1 | 1/1 | 1/1 | -| `robot_practice` | 10/11 | 9/9 | 8/8 | 10/11 | -| `sample_handling` | 2/2 | 2/2 | 2/2 | 2/2 | -| `sample_mapping` | 1/1 | 1/1 | 1/1 | 1/1 | -| `soc_recovery` | 2/2 | 2/2 | 1/1 | 2/2 | -| `supernatant_and_washes` | 1/1 | 1/1 | 1/1 | 1/1 | -| `template_and_primers` | 1/1 | 1/1 | 1/1 | 1/1 | -| `thermocycling` | 2/2 | 2/2 | 2/2 | 2/2 | -| `tips_and_contamination` | 11/11 | 9/9 | 8/8 | 11/11 | -| `volumes_and_wells` | 2/2 | 2/2 | 2/2 | 2/2 | -| `washes_and_drying` | 2/2 | 2/2 | 2/2 | 2/2 | - -Judge: openrouter anthropic/claude-sonnet-5.5 (x39) +| Rubric item | claude-sonnet-5.5 | claude-opus-5.5 | claude-fable-5.1 | gpt-6.1-sol | qwen3.8-2.4t-a95b | deepseek-v4-pro-0813 | +|---|---|---|---|---|---|---| +| `assembly_mix` | 2/2 | - | - | 2/2 | 2/2 | 2/2 | +| `binding` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `binding_and_separation` | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `cycling_and_transformation` | 2/2 | - | - | 2/2 | 2/2 | 2/2 | +| `deck_and_hardware` | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `dna_addition` | 2/2 | 2/2 | 1/1 | 2/2 | 2/2 | 2/2 | +| `dpni_and_cleanup` | 2/2 | - | - | 2/2 | 2/2 | 2/2 | +| `drying_and_elution` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `elution_recovery` | 2/2 | 2/2 | 2/2 | 2/2 | 1/2 | 1/2 | +| `fidelity_to_paper` | 2/4 | 2/3 | 2/2 | 1/4 | 0/4 | 2/4 | +| `fidelity_to_task` | 6/7 | 6/6 | 5/6 | 7/7 | 6/7 | 6/7 | +| `heat_shock` | 2/2 | 2/2 | 1/1 | 2/2 | 2/2 | 2/2 | +| `master_mix` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `pcr_setup` | 2/2 | - | - | 2/2 | 2/2 | 2/2 | +| `reaction_setup` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `recovery` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `robot_practice` | 10/11 | 9/9 | 8/8 | 10/11 | 10/11 | 10/11 | +| `sample_handling` | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `sample_mapping` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `soc_recovery` | 2/2 | 2/2 | 1/1 | 2/2 | 2/2 | 2/2 | +| `supernatant_and_washes` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `template_and_primers` | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | +| `thermocycling` | 2/2 | 2/2 | 2/2 | 2/2 | 1/2 | 2/2 | +| `tips_and_contamination` | 11/11 | 9/9 | 8/8 | 11/11 | 9/11 | 8/11 | +| `volumes_and_wells` | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | +| `washes_and_drying` | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | + +Judge: openrouter anthropic/claude-sonnet-5.5 (x61) ## Every failure, with the ground-truth evidence -**colony-pcr-screening-hard · claude-fable-5.1** (reward 1.00) -- check `end_state:pcr_plate` failed: expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20 - -**colony-pcr-screening-hard · claude-opus-5.5** (reward 1.00) -- check `end_state:pcr_plate` failed: expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20 +**ampure-bead-cleanup · deepseek-v4-pro-0813** (reward 0.30) +- check `no_cross_contamination` failed: [(23, 'A1'), (24, 'B1'), (25, 'B1')] +- judge 0 on `tips_and_contamination` (votes [0, 0, 0]): Steps 1, 11 and 13 use new_tip='once' with mix_after or a sample-to-sample transfer, so one tip touches many samples. It also returns to the bead and water stocks after mixing in samples. The no_cross_contamination check failed. +- judge 0 on `fidelity_to_task` (votes [0, 0, 0]): Shared tips across samples in steps 1, 11 and 13 cross-contaminate. The elution mix volume was changed to 30 µL. The protocol is therefore not fully faithful. **colony-pcr-screening-hard · claude-sonnet-5.5** (reward 0.75) -- check `end_state:pcr_plate` failed: expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20 -- judge 0 on `fidelity_to_paper` (votes [1, 0]): Thermocycling is only a comment with an annealing range (55-65 C) and 20-30 s/kb rather than a definite program. The 3 min initial denaturation and the assumed pre-diluted 4 uL primer are also unspecified adaptations. -- judge: 2 of 3 votes returned (network errors) +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The thermocycling comment leaves annealing as a range (55-65 C) and extension as 20-30 s/kb rather than a committed program. Primer is added before template and a 4 µL primer volume assumes a pre-diluted stock, so I cannot call it fully faithful. **colony-pcr-screening-hard · gpt-6.1-sol** (reward 0.50) -- check `end_state:pcr_plate` failed: expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20 -- judge 0 on `robot_practice` (votes [0, 0, 0]): The protocol adds a needless manual well-by-well transfer of all 96 reactions to a different plate plus several pauses, extra handling that puts samples at risk. Otherwise, apiLevel 2.15 is valid and all volumes (1-5 uL) are handled by the p20. -- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The protocol invents a manual transfer to a real PCR plate and extra pauses, and assumes a 1.25 uM primer concentration. The paper has no such steps, so it is not fully faithful. +- judge 0 on `robot_practice` (votes [0, 0, 0]): Valid apiLevel 2.15 and P20 volumes of 1–5 µL are fine, but the protocol tells the user to manually transfer all 96 reactions from the Corning pcr_plate into a different PCR plate. That is needless extra handling that puts samples at risk. +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The protocol invents a manual transfer into a separate 0.2 mL PCR plate, a gel electrophoresis step, and assumed primer dilutions and cycling values. The paper's OT-2 workflow runs the reaction in the destination plate. + +**colony-pcr-screening-hard · qwen3.8-2.4t-a95b** (reward 0.67) +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The protocol adds steps the paper does not have (mix and touch_tip). It also invents assumptions about primer Tm and concentration, and it describes an 'ice-chilled deck' that the code does not provide. Cycling uses a combined 72 C anneal/extend with a '4-10 C' hold instead of the paper-based adjust +- judge 0 on `thermocycling` (votes [0, 1, 0]): Sealing and cycling are recorded only in a final comment. The program is a two-step 98/72 C cycle with no separate annealing step. It rests on an invented assumption (Tm >= 69 C) and ends with a '4-10 C' hold rather than 4 C. **ecoli-heat-shock-transformation-hard · claude-fable-5.1** (reward 0.00) - AgentSafetyRefusalError @@ -110,28 +112,49 @@ Judge: openrouter anthropic/claude-sonnet-5.5 (x39) - judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): Adds steps the paper does not describe (p20.mix 3x5 µL on competent cells, p300.mix 3x30 µL, blow_outs) and an arbitrary 45 C lid setting, so it is not a fully faithful implementation. **ecoli-heat-shock-transformation-hard · claude-sonnet-5.5** (reward 0.75) -- judge 0 on `fidelity_to_paper` (votes [0]): The code adds a mixing step (p20.mix(3, 5, ...)) that the paper does not describe and sets the lid to 42 C, an invented choice. It also has no return to 4 C between the shock and SOC addition. -- judge: 1 of 3 votes returned (network errors) +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): Adds an invented p20.mix(3, 5) step after each DNA addition and sets the lid to 42 C, neither of which is in the paper, so not fully faithful. + +**ecoli-heat-shock-transformation-hard · deepseek-v4-pro-0813** (reward 0.75) +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): After the 42 °C step there is no return to 4 °C, so the block stays at 42 °C while the lid opens and 8 SOC transfers with mixing run. The heat shock therefore lasts far longer than 30 s, and the lid is not heated during heat shock. **ecoli-heat-shock-transformation-hard · gpt-6.1-sol** (reward 0.75) - judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The agent added steps the paper does not specify: p20.mix(3,5) in the cells after DNA addition and p300.mix(3,40) after SOC. It also deactivated the lid during the 4 C and 42 C steps and heated it to 50 C only for recovery. +**ecoli-heat-shock-transformation-hard · qwen3.8-2.4t-a95b** (reward 0.75) +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The protocol adds steps the paper does not describe: p20.mix(3,5) with blow_out after DNA addition, p300.mix(3,40) after SOC, and no lid heating, so it is not a fully faithful implementation. + **golden-gate-assembly · claude-fable-5.1** (reward 0.00) - AgentSafetyRefusalError **golden-gate-assembly · claude-opus-5.5** (reward 0.00) - AgentSafetyRefusalError +**golden-gate-assembly · qwen3.8-2.4t-a95b** (reward 0.30) +- check `no_cross_contamination` failed: [(379, 'D2'), (380, 'B1'), (381, 'B1')] +- judge 0 on `tips_and_contamination` (votes [0, 0, 0]): Measured check no_cross_contamination failed at (379,'D2') etc. Step 24 uses new_tip='once' with mix_after=(5,15), so the tip that mixed in assembly wells goes back into the gg_enzyme stock D2. +- judge 0 on `fidelity_to_task` (votes [0, 0, 0]): Step 24 reuses one tip with mix_after, which contaminates the enzyme stock and so deviates from a clean implementation. The mix volume of 15 µL is also arbitrary for 20 µL reactions. + **golden-gate-assembly-hard · claude-fable-5.1** (reward 0.00) - AgentSafetyRefusalError **golden-gate-assembly-hard · claude-opus-5.5** (reward 0.00) - AgentSafetyRefusalError +**golden-gate-assembly-hard · deepseek-v4-pro-0813** (reward 0.30) +- check `no_cross_contamination` failed: [(363, 'D2'), (364, 'B1'), (365, 'B1')] +- judge 0 on `tips_and_contamination` (votes [0, 0, 0]): no_cross_contamination failed at (363,'D2'),(364,'B1'),(365,'B1'): p20.transfer(1.0, gg_enzyme, assembly_wells, new_tip='once', mix_after=(3,15.0)) re-aspirates enzyme stock with a tip that has mixed in assembly wells. + **golden-gate-assembly-hard · gpt-6.1-sol** (reward 0.75) - check `end_state:pcr_plate` failed: expected {'A1': 48.0, 'B1': 38.0, 'C1': 48.0, 'D1': 48.0, 'E1': 42.0, 'F1': 38.0, 'G1': 48.0}, got {'A1': 49.0, 'B1': 39.0, 'C1': 49.0, 'D1': 49.0, 'E1': 43.0, 'F1': 39.0, 'G1': 49.0} - judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The agent adds an invented 1 µL water 'replace QC aliquot' step before DpnI. The end_state:pcr_plate check fails (49 vs 48 µL expected), so the protocol is not fully faithful. +**golden-gate-assembly-hard · qwen3.8-2.4t-a95b** (reward 0.30) +- check `no_cross_contamination` failed: [(395, 'D2'), (396, 'B1'), (397, 'B1')] +- check `end_state:assembly_plate` failed: expected {'A1': 15.0, 'B1': 15.0, 'C1': 15.0, 'D1': 15.0}, got {'A1': 10.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0} +- check `end_state:cells_plate` failed: expected {'A1': 305.0, 'B1': 305.0, 'C1': 305.0, 'D1': 305.0}, got {'A1': 310.0, 'B1': 310.0, 'C1': 310.0, 'D1': 310.0} +- judge 0 on `tips_and_contamination` (votes [0, 0, 0]): The no_cross_contamination check failed [(395,'D2'),(396,'B1'),(397,'B1')]: the gg_enzyme transfer uses new_tip='once' with mix_after=(5,15), so one tip mixes in assembly wells and then returns to the enzyme stock D2 and the next wells. +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The protocol deviates from the paper: fragments are eluted in 15 µL instead of 10 µL, the full 10 µL assembly eluate is transformed (the paper uses 5 µL), and the enzyme tip is reused across assembly wells; the end_state checks for assembly_plate and cells_plate also differ. + **opentrons-rna-extraction · claude-fable-5.1** (reward 0.75) - judge 0 on `fidelity_to_task` (votes [0, 0, 0]): The module docstring and metadata credit the paper to 'Lopez-Fernandez et al.', but the authors are Lázaro-Perona et al., so the metadata is inaccurate. The code also adds an unspecified bead-resuspension mix (m300.mix(5,150,beads...)). @@ -140,11 +163,20 @@ Judge: openrouter anthropic/claude-sonnet-5.5 (x39) - judge 0 on `robot_practice` (votes [0, 0, 0]): Check reservoir_columns_within_15ml failed: etoh[i % 4] draws 16000 µL from columns 9 and 10 (nest 12-well ~15 mL), which exceeds the well capacity. - judge 0 on `fidelity_to_task` (votes [1, 0, 0]): The ethanol allocation overdraws reservoir columns 9 and 10 (16000 µL each), so the run as coded cannot be carried out with the specified reservoir quantities. -**opentrons-rna-extraction-hard · claude-sonnet-5.5** (reward 0.69) -- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): Recovers 100 µL (p300m.aspirate(ELUTION_VOL...) in recovery loop) rather than ~80 µL, aspirating the full eluate near beads at 0.8 mm; also adds unspecified bead premix and elution mixing, so it is not fully faithful. -- judge 0 on `elution_recovery` (votes [0, 0, 0]): Elution 100 µL off-magnet with mix, 30 s, magnet 90 s are correct, but recovery aspirates and dispenses ELUTION_VOL=100 µL rather than about 80 µL. +**opentrons-rna-extraction · deepseek-v4-pro-0813** (reward 0.30) +- check `reservoir_columns_within_15ml` failed: uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 16000, 10: 16000, 11: 8000, 12: 8000 +- judge 0 on `robot_practice` (votes [0, 0, 0]): Check reservoir_columns_within_15ml failed: ethanol columns 9 and 10 are each drawn 16000 uL (ethanol[i % 4] sends two columns of wash volume to each), exceeding the 15 mL well capacity. + +**opentrons-rna-extraction · qwen3.8-2.4t-a95b** (reward 0.30) +- check `reservoir_columns_within_15ml` failed: uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 12000, 10: 16000, 11: 12000, 12: 8000 +- judge 0 on `robot_practice` (votes [0, 0, 0]): Failed check reservoir_columns_within_15ml: the ethanol split (etoh_cols[i%4] and [(i+1)%4]) draws 16000 uL from reservoir column 10, more than the 15 mL well holds. + +**opentrons-rna-extraction-hard · deepseek-v4-pro-0813** (reward 0.44) +- judge 0 on `tips_and_contamination` (votes [0, 0, 0]): One p1000 tip is reused for 48 wells for isopropanol and for both 500 µL ethanol dispenses at default well-bottom height into sample wells, and one p300 tip dispenses elution buffer into all six columns, so tips contact sample-bead wells and then move to others. +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): The 100 µL elution buffer is dispensed with no mixing/resuspension of the beads, and shared tips are used across sample wells, so it is not a fully faithful, sound implementation. +- judge 0 on `elution_recovery` (votes [0, 0, 0]): Step 7 only aspirates/dispenses 100 µL elution buffer into each column with no mixing afterward, so the beads are not resuspended as the rubric requires, though 90 s magnet and 100 µL recovery to distinct wells pass. -**opentrons-rna-extraction-hard · gpt-6.1-sol** (reward 0.69) -- judge 0 on `fidelity_to_paper` (votes [0, 1, 0]): The eluate recovery aspirates and moves the full ELUTION_VOLUME (100 µL) from a well holding 100 µL, with the magnet engaged. The expected recovery is about 80 µL, so this deviates from the specification. -- judge 0 on `elution_recovery` (votes [0, 1, 0]): Recovery is the full 100 µL (aspirate ELUTION_VOLUME from source.bottom(z=0.5)), not about 80 µL. This risks drawing beads and does not match the expected recovery volume. +**opentrons-rna-extraction-hard · qwen3.8-2.4t-a95b** (reward 0.69) +- judge 0 on `fidelity_to_paper` (votes [0, 0, 0]): No mixing after elution buffer addition, and only 500 µL of the 540 µL supernatant is removed, leaving about 40 µL. Both are deviations or additions, and this item is all-or-nothing. +- judge 0 on `elution_recovery` (votes [0, 0, 0]): Elution buffer is dispensed with the magnet off but never mixed. There is only a 30 s delay, then the magnet is engaged, so the rubric's 'added off-magnet and mixed' is not met, although recovery of 100 µL to distinct wells is fine. diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/a1-a12-100ul/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/a1-a12-100ul/trial.json index 7a57c72..646f131 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/a1-a12-100ul/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/a1-a12-100ul/trial.json @@ -38,5 +38,6 @@ "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0 - } + }, + "cache_tokens": 45309 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/ampure-bead-cleanup/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/ampure-bead-cleanup/trial.json index d4ca37c..dcad123 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/ampure-bead-cleanup/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/ampure-bead-cleanup/trial.json @@ -42,5 +42,6 @@ "rubric_supernatant_and_washes": 1.0, "rubric_drying_and_elution": 1.0, "rubric_recovery": 1.0 - } + }, + "cache_tokens": 73733 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/grader.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/grader.json index 817df1b..9aa2152 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/grader.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/grader.json @@ -42,9 +42,9 @@ "detail": "" }, { - "name": "end_state:pcr_plate", - "pass": false, - "detail": "expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20.0, 'C1': 20.0, 'C2': 20.0, 'C3': 20.0, 'C4': 20.0, 'C5': 20.0, 'C6': 20.0, 'C7': 20.0, 'C8': 20.0, 'C9': 20.0, 'C10': 20.0, 'C11': 20.0, 'C12': 20.0, 'D1': 20.0, 'D2': 20.0, 'D3': 20.0, 'D4': 20.0, 'D5': 20.0, 'D6': 20.0, 'D7': 20.0, 'D8': 20.0, 'D9': 20.0, 'D10': 20.0, 'D11': 20.0, 'D12': 20.0, 'E1': 20.0, 'E2': 20.0, 'E3': 20.0, 'E4': 20.0, 'E5': 20.0, 'E6': 20.0, 'E7': 20.0, 'E8': 20.0, 'E9': 20.0, 'E10': 20.0, 'E11': 20.0, 'E12': 20.0, 'F1': 20.0, 'F2': 20.0, 'F3': 20.0, 'F4': 20.0, 'F5': 20.0, 'F6': 20.0, 'F7': 20.0, 'F8': 20.0, 'F9': 20.0, 'F10': 20.0, 'F11': 20.0, 'F12': 20.0, 'G1': 20.0, 'G2': 20.0, 'G3': 20.0, 'G4': 20.0, 'G5': 20.0, 'G6': 20.0, 'G7': 20.0, 'G8': 20.0, 'G9': 20.0, 'G10': 20.0, 'G11': 20.0, 'G12': 20.0, 'H1': 20.0, 'H2': 20.0, 'H3': 20.0, 'H4': 20.0, 'H5': 20.0, 'H6': 20.0, 'H7': 20.0, 'H8': 20.0, 'H9': 20.0, 'H10': 20.0, 'H11': 20.0, 'H12': 20.0}, got {'A1': 10.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0, 'E1': 10.0, 'F1': 10.0, 'G1': 10.0, 'H1': 10.0, 'A2': 10.0, 'B2': 10.0, 'C2': 10.0, 'D2': 10.0, 'E2': 10.0, 'F2': 10.0, 'G2': 10.0, 'H2': 10.0, 'A3': 10.0, 'B3': 10.0, 'C3': 10.0, 'D3': 10.0, 'E3': 10.0, 'F3': 10.0, 'G3': 10.0, 'H3': 10.0, 'A4': 10.0, 'B4': 10.0, 'C4': 10.0, 'D4': 10.0, 'E4': 10.0, 'F4': 10.0, 'G4': 10.0, 'H4': 10.0, 'A5': 10.0, 'B5': 10.0, 'C5': 10.0, 'D5': 10.0, 'E5': 10.0, 'F5': 10.0, 'G5': 10.0, 'H5': 10.0, 'A6': 10.0, 'B6': 10.0, 'C6': 10.0, 'D6': 10.0, 'E6': 10.0, 'F6': 10.0, 'G6': 10.0, 'H6': 10.0, 'A7': 10.0, 'B7': 10.0, 'C7': 10.0, 'D7': 10.0, 'E7': 10.0, 'F7': 10.0, 'G7': 10.0, 'H7': 10.0, 'A8': 10.0, 'B8': 10.0, 'C8': 10.0, 'D8': 10.0, 'E8': 10.0, 'F8': 10.0, 'G8': 10.0, 'H8': 10.0, 'A9': 10.0, 'B9': 10.0, 'C9': 10.0, 'D9': 10.0, 'E9': 10.0, 'F9': 10.0, 'G9': 10.0, 'H9': 10.0, 'A10': 10.0, 'B10': 10.0, 'C10': 10.0, 'D10': 10.0, 'E10': 10.0, 'F10': 10.0, 'G10': 10.0, 'H10': 10.0, 'A11': 10.0, 'B11': 10.0, 'C11': 10.0, 'D11': 10.0, 'E11': 10.0, 'F11': 10.0, 'G11': 10.0, 'H11': 10.0, 'A12': 10.0, 'B12': 10.0, 'C12': 10.0, 'D12': 10.0, 'E12': 10.0, 'F12': 10.0, 'G12': 10.0, 'H12': 10.0}" + "name": "composition:pcr_plate", + "pass": true, + "detail": "" }, { "name": "end_state:primer_plate", @@ -155,5 +155,8 @@ "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0 - } + }, + "not_applicable": [ + "end_state:pcr_plate (the source paper does not determine the exact volumes; composition checked instead)" + ] } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/reward.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/reward.json index 903e0a1..6524a91 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/reward.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/reward.json @@ -1 +1 @@ -{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 0.8889, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.3333, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 1.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 1.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/trial.json index 5827c35..957b013 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening-hard/trial.json @@ -6,10 +6,10 @@ "rewards": { "reward": 1.0, "sim_pass": 1.0, - "checks_frac": 0.8889, + "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, - "deterministic_reward": 0.3333, + "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, @@ -40,5 +40,6 @@ "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0 - } + }, + "cache_tokens": 273978 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening/trial.json index 749330d..ca2d90e 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/colony-pcr-screening/trial.json @@ -40,5 +40,6 @@ "rubric_master_mix": 1.0, "rubric_template_and_primers": 1.0, "rubric_thermocycling": 1.0 - } + }, + "cache_tokens": 99413 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation-hard/trial.json index fecfab3..3bba6cf 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation-hard/trial.json @@ -14,5 +14,6 @@ "input_tokens": 0, "output_tokens": 0, "started_at": "2026-10-07T01:46:26.563009Z", - "finished_at": "2026-10-07T01:46:51.933255Z" + "finished_at": "2026-10-07T01:46:51.933255Z", + "cache_tokens": 0 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation/trial.json index 9347810..b90eb93 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/ecoli-heat-shock-transformation/trial.json @@ -40,5 +40,6 @@ "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0 - } + }, + "cache_tokens": 70645 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly-hard/trial.json index 887a866..b530ceb 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly-hard/trial.json @@ -14,5 +14,6 @@ "input_tokens": 0, "output_tokens": 0, "started_at": "2026-10-07T01:20:47.021285Z", - "finished_at": "2026-10-07T01:21:58.980708Z" + "finished_at": "2026-10-07T01:21:58.980708Z", + "cache_tokens": 0 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly/trial.json index 2af210d..7aab49a 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/golden-gate-assembly/trial.json @@ -14,5 +14,6 @@ "input_tokens": 0, "output_tokens": 0, "started_at": "2026-10-07T01:20:47.064478Z", - "finished_at": "2026-10-07T01:22:02.329073Z" + "finished_at": "2026-10-07T01:22:02.329073Z", + "cache_tokens": 0 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction-hard/trial.json index d64665b..b6316b0 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction-hard/trial.json @@ -44,5 +44,6 @@ "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0 - } + }, + "cache_tokens": 484038 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction/trial.json index 4c5a36b..289d688 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/opentrons-rna-extraction/trial.json @@ -44,5 +44,6 @@ "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0 - } + }, + "cache_tokens": 219941 } diff --git a/results/runs/2026-10-07-openrouter/claude-fable-5.1/split-200ul-two-wells/trial.json b/results/runs/2026-10-07-openrouter/claude-fable-5.1/split-200ul-two-wells/trial.json index 946a320..83aa82a 100644 --- a/results/runs/2026-10-07-openrouter/claude-fable-5.1/split-200ul-two-wells/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-fable-5.1/split-200ul-two-wells/trial.json @@ -38,5 +38,6 @@ "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0 - } + }, + "cache_tokens": 94973 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/a1-a12-100ul/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/a1-a12-100ul/trial.json index 06f0e0f..51eda92 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/a1-a12-100ul/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/a1-a12-100ul/trial.json @@ -38,5 +38,6 @@ "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0 - } + }, + "cache_tokens": 43930 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/ampure-bead-cleanup/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/ampure-bead-cleanup/trial.json index acbcd58..34c55e5 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/ampure-bead-cleanup/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/ampure-bead-cleanup/trial.json @@ -42,5 +42,6 @@ "rubric_supernatant_and_washes": 1.0, "rubric_drying_and_elution": 1.0, "rubric_recovery": 1.0 - } + }, + "cache_tokens": 99909 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/grader.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/grader.json index 81d8198..ed2c778 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/grader.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/grader.json @@ -42,9 +42,9 @@ "detail": "" }, { - "name": "end_state:pcr_plate", - "pass": false, - "detail": "expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20.0, 'C1': 20.0, 'C2': 20.0, 'C3': 20.0, 'C4': 20.0, 'C5': 20.0, 'C6': 20.0, 'C7': 20.0, 'C8': 20.0, 'C9': 20.0, 'C10': 20.0, 'C11': 20.0, 'C12': 20.0, 'D1': 20.0, 'D2': 20.0, 'D3': 20.0, 'D4': 20.0, 'D5': 20.0, 'D6': 20.0, 'D7': 20.0, 'D8': 20.0, 'D9': 20.0, 'D10': 20.0, 'D11': 20.0, 'D12': 20.0, 'E1': 20.0, 'E2': 20.0, 'E3': 20.0, 'E4': 20.0, 'E5': 20.0, 'E6': 20.0, 'E7': 20.0, 'E8': 20.0, 'E9': 20.0, 'E10': 20.0, 'E11': 20.0, 'E12': 20.0, 'F1': 20.0, 'F2': 20.0, 'F3': 20.0, 'F4': 20.0, 'F5': 20.0, 'F6': 20.0, 'F7': 20.0, 'F8': 20.0, 'F9': 20.0, 'F10': 20.0, 'F11': 20.0, 'F12': 20.0, 'G1': 20.0, 'G2': 20.0, 'G3': 20.0, 'G4': 20.0, 'G5': 20.0, 'G6': 20.0, 'G7': 20.0, 'G8': 20.0, 'G9': 20.0, 'G10': 20.0, 'G11': 20.0, 'G12': 20.0, 'H1': 20.0, 'H2': 20.0, 'H3': 20.0, 'H4': 20.0, 'H5': 20.0, 'H6': 20.0, 'H7': 20.0, 'H8': 20.0, 'H9': 20.0, 'H10': 20.0, 'H11': 20.0, 'H12': 20.0}, got {'A1': 10.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0, 'E1': 10.0, 'F1': 10.0, 'G1': 10.0, 'H1': 10.0, 'A2': 10.0, 'B2': 10.0, 'C2': 10.0, 'D2': 10.0, 'E2': 10.0, 'F2': 10.0, 'G2': 10.0, 'H2': 10.0, 'A3': 10.0, 'B3': 10.0, 'C3': 10.0, 'D3': 10.0, 'E3': 10.0, 'F3': 10.0, 'G3': 10.0, 'H3': 10.0, 'A4': 10.0, 'B4': 10.0, 'C4': 10.0, 'D4': 10.0, 'E4': 10.0, 'F4': 10.0, 'G4': 10.0, 'H4': 10.0, 'A5': 10.0, 'B5': 10.0, 'C5': 10.0, 'D5': 10.0, 'E5': 10.0, 'F5': 10.0, 'G5': 10.0, 'H5': 10.0, 'A6': 10.0, 'B6': 10.0, 'C6': 10.0, 'D6': 10.0, 'E6': 10.0, 'F6': 10.0, 'G6': 10.0, 'H6': 10.0, 'A7': 10.0, 'B7': 10.0, 'C7': 10.0, 'D7': 10.0, 'E7': 10.0, 'F7': 10.0, 'G7': 10.0, 'H7': 10.0, 'A8': 10.0, 'B8': 10.0, 'C8': 10.0, 'D8': 10.0, 'E8': 10.0, 'F8': 10.0, 'G8': 10.0, 'H8': 10.0, 'A9': 10.0, 'B9': 10.0, 'C9': 10.0, 'D9': 10.0, 'E9': 10.0, 'F9': 10.0, 'G9': 10.0, 'H9': 10.0, 'A10': 10.0, 'B10': 10.0, 'C10': 10.0, 'D10': 10.0, 'E10': 10.0, 'F10': 10.0, 'G10': 10.0, 'H10': 10.0, 'A11': 10.0, 'B11': 10.0, 'C11': 10.0, 'D11': 10.0, 'E11': 10.0, 'F11': 10.0, 'G11': 10.0, 'H11': 10.0, 'A12': 10.0, 'B12': 10.0, 'C12': 10.0, 'D12': 10.0, 'E12': 10.0, 'F12': 10.0, 'G12': 10.0, 'H12': 10.0}" + "name": "composition:pcr_plate", + "pass": true, + "detail": "" }, { "name": "end_state:primer_plate", @@ -155,5 +155,8 @@ "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0 - } + }, + "not_applicable": [ + "end_state:pcr_plate (the source paper does not determine the exact volumes; composition checked instead)" + ] } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/reward.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/reward.json index 903e0a1..6524a91 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/reward.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/reward.json @@ -1 +1 @@ -{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 0.8889, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.3333, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 1.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 1.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/trial.json index 4731505..5c26cc0 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening-hard/trial.json @@ -6,10 +6,10 @@ "rewards": { "reward": 1.0, "sim_pass": 1.0, - "checks_frac": 0.8889, + "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, - "deterministic_reward": 0.3333, + "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, @@ -40,5 +40,6 @@ "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0 - } + }, + "cache_tokens": 211127 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening/trial.json index 891d925..153976c 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/colony-pcr-screening/trial.json @@ -40,5 +40,6 @@ "rubric_master_mix": 1.0, "rubric_template_and_primers": 1.0, "rubric_thermocycling": 1.0 - } + }, + "cache_tokens": 100678 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation-hard/trial.json index 0152847..563152d 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation-hard/trial.json @@ -40,5 +40,6 @@ "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0 - } + }, + "cache_tokens": 153416 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation/trial.json index 5cff0c2..0b4c03b 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/ecoli-heat-shock-transformation/trial.json @@ -40,5 +40,6 @@ "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0 - } + }, + "cache_tokens": 69576 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly-hard/trial.json index da261bb..24b91fa 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly-hard/trial.json @@ -14,5 +14,6 @@ "input_tokens": 0, "output_tokens": 0, "started_at": "2026-10-07T01:12:04.721732Z", - "finished_at": "2026-10-07T01:13:12.873896Z" + "finished_at": "2026-10-07T01:13:12.873896Z", + "cache_tokens": 0 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly/trial.json index b7c89b3..a09f4a6 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/golden-gate-assembly/trial.json @@ -14,5 +14,6 @@ "input_tokens": 0, "output_tokens": 0, "started_at": "2026-10-07T01:12:04.776078Z", - "finished_at": "2026-10-07T01:13:12.884803Z" + "finished_at": "2026-10-07T01:13:12.884803Z", + "cache_tokens": 0 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction-hard/trial.json index ea6bf7e..e03f49a 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction-hard/trial.json @@ -44,5 +44,6 @@ "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 0.0 - } + }, + "cache_tokens": 526073 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction/trial.json index 61b772b..10905e3 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/opentrons-rna-extraction/trial.json @@ -44,5 +44,6 @@ "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0 - } + }, + "cache_tokens": 283526 } diff --git a/results/runs/2026-10-07-openrouter/claude-opus-5.5/split-200ul-two-wells/trial.json b/results/runs/2026-10-07-openrouter/claude-opus-5.5/split-200ul-two-wells/trial.json index db5c0d6..4915e8b 100644 --- a/results/runs/2026-10-07-openrouter/claude-opus-5.5/split-200ul-two-wells/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-opus-5.5/split-200ul-two-wells/trial.json @@ -38,5 +38,6 @@ "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0 - } + }, + "cache_tokens": 68423 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/a1-a12-100ul/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/a1-a12-100ul/trial.json index 5eebbbb..6d97924 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/a1-a12-100ul/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/a1-a12-100ul/trial.json @@ -38,5 +38,6 @@ "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0 - } + }, + "cache_tokens": 67730 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ampure-bead-cleanup/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ampure-bead-cleanup/trial.json index af073a6..1e68e29 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ampure-bead-cleanup/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ampure-bead-cleanup/trial.json @@ -42,5 +42,6 @@ "rubric_supernatant_and_washes": 1.0, "rubric_drying_and_elution": 1.0, "rubric_recovery": 1.0 - } + }, + "cache_tokens": 68689 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/grader.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/grader.json index 1c8831d..b528bb4 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/grader.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/grader.json @@ -5,6 +5,9 @@ "ok": true, "n_commands": 2280 }, + "not_applicable": [ + "end_state:pcr_plate (the source paper does not determine the exact volumes; composition checked instead)" + ], "checks": [ { "name": "simulator_ran", @@ -42,9 +45,9 @@ "detail": "" }, { - "name": "end_state:pcr_plate", - "pass": false, - "detail": "expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20.0, 'C1': 20.0, 'C2': 20.0, 'C3': 20.0, 'C4': 20.0, 'C5': 20.0, 'C6': 20.0, 'C7': 20.0, 'C8': 20.0, 'C9': 20.0, 'C10': 20.0, 'C11': 20.0, 'C12': 20.0, 'D1': 20.0, 'D2': 20.0, 'D3': 20.0, 'D4': 20.0, 'D5': 20.0, 'D6': 20.0, 'D7': 20.0, 'D8': 20.0, 'D9': 20.0, 'D10': 20.0, 'D11': 20.0, 'D12': 20.0, 'E1': 20.0, 'E2': 20.0, 'E3': 20.0, 'E4': 20.0, 'E5': 20.0, 'E6': 20.0, 'E7': 20.0, 'E8': 20.0, 'E9': 20.0, 'E10': 20.0, 'E11': 20.0, 'E12': 20.0, 'F1': 20.0, 'F2': 20.0, 'F3': 20.0, 'F4': 20.0, 'F5': 20.0, 'F6': 20.0, 'F7': 20.0, 'F8': 20.0, 'F9': 20.0, 'F10': 20.0, 'F11': 20.0, 'F12': 20.0, 'G1': 20.0, 'G2': 20.0, 'G3': 20.0, 'G4': 20.0, 'G5': 20.0, 'G6': 20.0, 'G7': 20.0, 'G8': 20.0, 'G9': 20.0, 'G10': 20.0, 'G11': 20.0, 'G12': 20.0, 'H1': 20.0, 'H2': 20.0, 'H3': 20.0, 'H4': 20.0, 'H5': 20.0, 'H6': 20.0, 'H7': 20.0, 'H8': 20.0, 'H9': 20.0, 'H10': 20.0, 'H11': 20.0, 'H12': 20.0}, got {'A1': 10.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0, 'E1': 10.0, 'F1': 10.0, 'G1': 10.0, 'H1': 10.0, 'A2': 10.0, 'B2': 10.0, 'C2': 10.0, 'D2': 10.0, 'E2': 10.0, 'F2': 10.0, 'G2': 10.0, 'H2': 10.0, 'A3': 10.0, 'B3': 10.0, 'C3': 10.0, 'D3': 10.0, 'E3': 10.0, 'F3': 10.0, 'G3': 10.0, 'H3': 10.0, 'A4': 10.0, 'B4': 10.0, 'C4': 10.0, 'D4': 10.0, 'E4': 10.0, 'F4': 10.0, 'G4': 10.0, 'H4': 10.0, 'A5': 10.0, 'B5': 10.0, 'C5': 10.0, 'D5': 10.0, 'E5': 10.0, 'F5': 10.0, 'G5': 10.0, 'H5': 10.0, 'A6': 10.0, 'B6': 10.0, 'C6': 10.0, 'D6': 10.0, 'E6': 10.0, 'F6': 10.0, 'G6': 10.0, 'H6': 10.0, 'A7': 10.0, 'B7': 10.0, 'C7': 10.0, 'D7': 10.0, 'E7': 10.0, 'F7': 10.0, 'G7': 10.0, 'H7': 10.0, 'A8': 10.0, 'B8': 10.0, 'C8': 10.0, 'D8': 10.0, 'E8': 10.0, 'F8': 10.0, 'G8': 10.0, 'H8': 10.0, 'A9': 10.0, 'B9': 10.0, 'C9': 10.0, 'D9': 10.0, 'E9': 10.0, 'F9': 10.0, 'G9': 10.0, 'H9': 10.0, 'A10': 10.0, 'B10': 10.0, 'C10': 10.0, 'D10': 10.0, 'E10': 10.0, 'F10': 10.0, 'G10': 10.0, 'H10': 10.0, 'A11': 10.0, 'B11': 10.0, 'C11': 10.0, 'D11': 10.0, 'E11': 10.0, 'F11': 10.0, 'G11': 10.0, 'H11': 10.0, 'A12': 10.0, 'B12': 10.0, 'C12': 10.0, 'D12': 10.0, 'E12': 10.0, 'F12': 10.0, 'G12': 10.0, 'H12': 10.0}" + "name": "composition:pcr_plate", + "pass": true, + "detail": "" }, { "name": "end_state:primer_plate", @@ -58,8 +61,9 @@ { "id": "robot_practice", "score": 1, - "evidence": "apiLevel 2.15; all transfers are 1\u201315 \u00b5L on the p20; the 10 \u00b5L final volume is far below well capacity; labware is used as intended; the simulator passed.", + "evidence": "apiLevel 2.15; all volumes (1\u201315 \u00b5L aspirates, 5/4/1 \u00b5L dispenses) are handled by the p20; the 10 \u00b5L final volume is well within well capacity; all measured checks pass.", "votes": [ + 1, 1, 1 ] @@ -67,8 +71,9 @@ { "id": "tips_and_contamination", "score": 1, - "evidence": "One tip is used for master mix into empty wells without touching them; a fresh tip is used for every primer pair and every colony; tip_before_aspirate, no_cross_contamination and tip_dropped_at_end all pass.", + "evidence": "One tip for master mix dispensed with no well contact, then a fresh tip per primer pair and per colony; every tip is dropped (tip_dropped_at_end and no_cross_contamination pass).", "votes": [ + 1, 1, 1 ] @@ -76,17 +81,19 @@ { "id": "fidelity_to_paper", "score": 0, - "evidence": "Thermocycling is only a comment with an annealing range (55-65 C) and 20-30 s/kb rather than a definite program. The 3 min initial denaturation and the assumed pre-diluted 4 uL primer are also unspecified adaptations.", + "evidence": "The thermocycling comment leaves annealing as a range (55-65 C) and extension as 20-30 s/kb rather than a committed program. Primer is added before template and a 4 \u00b5L primer volume assumes a pre-diluted stock, so I cannot call it fully faithful.", "votes": [ - 1, + 0, + 0, 0 ] }, { "id": "reaction_setup", "score": 1, - "evidence": "MM_VOL=5, PRIMER_VOL=4, TEMPLATE_VOL=1 gives a 10 \u00b5L reaction with 1x master mix. Master mix goes in first, then primer, then 1 \u00b5L template.", + "evidence": "Each well gets 5 \u00b5L 2x Q5 (1x in 10 \u00b5L), 4 \u00b5L of its primer pair and 1 \u00b5L template; master mix goes in first and template last, as in the paper.", "votes": [ + 1, 1, 1 ] @@ -94,8 +101,9 @@ { "id": "sample_mapping", "score": 1, - "evidence": "zip(primer_plate.wells(), dest) and zip(colony_plate.wells(), dest) map each source well to the same-position PCR well, covering all 96 once.", + "evidence": "zip(primer_plate.wells(), dest) and zip(colony_plate.wells(), dest) map A1\u2192A1 through H12\u2192H12 for all 96 wells, each used once.", "votes": [ + 1, 1, 1 ] @@ -103,8 +111,9 @@ { "id": "thermocycling", "score": 1, - "evidence": "Comments record plate sealing and 98 C initial denaturation, 30 cycles of 98 C/annealing/72 C, a 72 C 2 min final extension and a 4 C hold, which suit Q5.", + "evidence": "Comments record plate sealing and a program of 98 C initial denaturation, 30 cycles of denature/anneal/extend, 72 C 2 min final extension and a 4 C hold, suited to Q5.", "votes": [ + 1, 1, 1 ] @@ -118,7 +127,7 @@ "sample_mapping": 1.0, "thermocycling": 1.0 }, - "summary": "The protocol builds 10 \u00b5L reactions in the paper's order (master mix, then primer, then template), uses fresh tips per sample, and maps all 96 wells correctly. It also records sealing and a Q5-appropriate cycling program, so all items pass. The thermocycling annealing temperature is only given as a range, which is a minor looseness.", + "summary": "The protocol builds correct 10 \u00b5L reactions with sound tip handling and a 1:1 well mapping, and all measured checks pass. Fidelity is failed because the cycling program is stated as ranges and not committed values, and the primer-before-template order and the assumed pre-diluted primer stock are small departures.", "level": "hard", "score": 0.75, "weights": { @@ -130,17 +139,17 @@ "thermocycling": 0.0833 }, "provider": "openrouter", - "votes": 2, - "failed_votes": 1 + "votes": 3, + "failed_votes": 0 }, "critical_failures": [], "rewards": { "reward": 0.75, "sim_pass": 1.0, - "checks_frac": 0.8889, + "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, - "deterministic_reward": 0.3333, + "deterministic_reward": 1.0, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/reward.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/reward.json index e1ca78a..c32e10c 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/reward.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/reward.json @@ -1 +1 @@ -{"reward": 0.75, "sim_pass": 1.0, "checks_frac": 0.8889, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.3333, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} +{"reward": 0.75, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/trial.json index f78baff..09c8362 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening-hard/trial.json @@ -6,10 +6,10 @@ "rewards": { "reward": 0.75, "sim_pass": 1.0, - "checks_frac": 0.8889, + "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, - "deterministic_reward": 0.3333, + "deterministic_reward": 1.0, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, @@ -40,5 +40,6 @@ "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0 - } + }, + "cache_tokens": 112113 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening/trial.json index 54e3b68..ecc100a 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/colony-pcr-screening/trial.json @@ -40,5 +40,6 @@ "rubric_master_mix": 1.0, "rubric_template_and_primers": 1.0, "rubric_thermocycling": 1.0 - } + }, + "cache_tokens": 113855 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/grader.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/grader.json index 7ae8b3c..b4c1e47 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/grader.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/grader.json @@ -5,6 +5,7 @@ "ok": true, "n_commands": 133 }, + "not_applicable": [], "checks": [ { "name": "simulator_ran", @@ -53,48 +54,60 @@ { "id": "robot_practice", "score": 1, - "evidence": "apiLevel 2.13, p20 used for 1 \u00b5L and p300 for 50 \u00b5L, the 3x5 \u00b5L mix stays within the 11 \u00b5L well volume, and all labware is used as intended; the simulator ran clean.", + "evidence": "apiLevel 2.13, p20 used for 1 \u00b5L and p300 for 50 \u00b5L, correct labware/module use, simulator and measured checks all pass.", "votes": [ + 1, + 1, 1 ] }, { "id": "tips_and_contamination", "score": 1, - "evidence": "Every plasmid and every SOC transfer uses its own pick_up_tip and drop_tip; the checks tip_before_aspirate, tip_dropped_at_end and no_cross_contamination all passed.", + "evidence": "Fresh pick_up_tip/drop_tip for each plasmid and each SOC dispense; tip_dropped_at_end and no_cross_contamination checks pass.", "votes": [ + 1, + 1, 1 ] }, { "id": "fidelity_to_paper", "score": 0, - "evidence": "The code adds a mixing step (p20.mix(3, 5, ...)) that the paper does not describe and sets the lid to 42 C, an invented choice. It also has no return to 4 C between the shock and SOC addition.", + "evidence": "Adds an invented p20.mix(3, 5) step after each DNA addition and sets the lid to 42 C, neither of which is in the paper, so not fully faithful.", "votes": [ + 0, + 0, 0 ] }, { "id": "dna_addition", "score": 1, - "evidence": "The block is set to 4 C with the lid open, then 1 \u00b5L is aspirated from plasmid_plate[w] and dispensed into tc_plate[w] for A1-H1 with a fresh tip each time.", + "evidence": "Aspirates 1 \u00b5L from plasmid_plate[w] and dispenses into tc_plate[w] for all 8 wells with the block set to 4 C and lid open.", "votes": [ + 1, + 1, 1 ] }, { "id": "heat_shock", "score": 1, - "evidence": "The lid is closed, then set_block_temperature(4, hold_time_minutes=30) and set_block_temperature(42, hold_time_seconds=30) run on the thermocycler.", + "evidence": "Lid closed, set_block_temperature(4, hold 30 min) then (42, hold 30 s) on the thermocycler.", "votes": [ + 1, + 1, 1 ] }, { "id": "soc_recovery", "score": 1, - "evidence": "50 \u00b5L SOC goes into each well with the p300 and a fresh tip, then the lid is closed and the block is held at 37 C for 60 min.", + "evidence": "Dispenses 50 \u00b5L SOC to each well with a fresh tip, then closes lid and holds 37 C for 60 min.", "votes": [ + 1, + 1, 1 ] } @@ -107,7 +120,7 @@ "heat_shock": 1.0, "soc_recovery": 1.0 }, - "summary": "The protocol runs correctly: tip handling, volumes and the 4 C, 42 C and 37 C thermocycler steps all match the paper, and all measured checks passed. It fails fidelity because it adds an unspecified mixing step and uses a 42 C lid temperature the paper does not give.", + "summary": "The protocol correctly implements the LV configuration, tip handling, DNA addition, heat shock, and SOC recovery on the thermocycler. It fails strict fidelity because of an invented mixing step and a non-paper lid temperature of 42 C.", "level": "hard", "score": 0.75, "weights": { @@ -119,8 +132,8 @@ "soc_recovery": 0.0833 }, "provider": "openrouter", - "votes": 1, - "failed_votes": 2 + "votes": 3, + "failed_votes": 0 }, "critical_failures": [], "rewards": { diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/trial.json index b362b40..f884963 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation-hard/trial.json @@ -40,5 +40,6 @@ "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0 - } + }, + "cache_tokens": 161938 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation/trial.json index feeb2a3..6404fd1 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/ecoli-heat-shock-transformation/trial.json @@ -40,5 +40,6 @@ "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0 - } + }, + "cache_tokens": 68100 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly-hard/trial.json index 2602685..5b98f48 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly-hard/trial.json @@ -42,5 +42,6 @@ "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 1.0, "rubric_cycling_and_transformation": 1.0 - } + }, + "cache_tokens": 168165 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly/trial.json index 034fd3e..e3d8c2e 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/golden-gate-assembly/trial.json @@ -42,5 +42,6 @@ "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 1.0, "rubric_cycling_and_transformation": 1.0 - } + }, + "cache_tokens": 119963 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/grader.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/grader.json index e21f8bd..8f27819 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/grader.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/grader.json @@ -118,7 +118,7 @@ { "id": "robot_practice", "score": 1, - "evidence": "apiLevel 2.13; all volumes within p300 multi (40, 125, ~167, 180, 100) and p1000 (250, mix 400) ranges; wells under capacity; reservoir columns within 15 mL per check; simulator passed.", + "evidence": "apiLevel 2.13; p1000 handles 250 uL and mix 400, p300 multi handles volumes \u2264200 uL via chunks (180 uL for 540 uL, 167 uL for 500 uL); wells stay under 2.4 mL and reservoir columns within 15 mL (check passed); 34 tip columns fit in the 36 available.", "votes": [ 1, 1, @@ -128,7 +128,7 @@ { "id": "tips_and_contamination", "score": 1, - "evidence": "Check fresh_tip_per_sample_no_cross_contact passed (48 distinct tips); every pick_up_tip is followed by drop_tip; reagent dispensed at col.top without touching sample liquid; fresh tips for removal, elution and recovery.", + "evidence": "Every pipetting action has a tip and every tip is dropped; check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips; removals and elution recovery use a fresh tip per column; shared reagent tips only dispense at well top(-2) before sample contact.", "votes": [ 1, 1, @@ -137,18 +137,18 @@ }, { "id": "fidelity_to_paper", - "score": 0, - "evidence": "Recovers 100 \u00b5L (p300m.aspirate(ELUTION_VOL...) in recovery loop) rather than ~80 \u00b5L, aspirating the full eluate near beads at 0.8 mm; also adds unspecified bead premix and elution mixing, so it is not fully faithful.", + "score": 1, + "evidence": "Order and numbers match the paper: 40 uL beads, 250 uL IPA, 250 uL sample, mix 5x, 5 min, magnet 4 min, two 500 uL ethanol washes, 4 min dry, elute 100 uL, 30 s, magnet 90 s, recover; metadata citation is accurate and all 17 checks passed.", "votes": [ - 0, - 0, - 0 + 1, + 1, + 1 ] }, { "id": "sample_handling", "score": 1, - "evidence": "Check 48_samples_to_odd_columns passed; sources zip to sample_wells built from odd columns 1,3,5,7,9,11; each sample maps to its own well and elution well.", + "evidence": "Checks '48_samples_to_odd_columns' and 'distinct_elution_wells' passed; sources rack1+rack2 map one-to-one onto odd-column wells with a fresh p1000 tip each.", "votes": [ 1, 1, @@ -158,7 +158,7 @@ { "id": "binding_and_separation", "score": 1, - "evidence": "Beads 40, IPA 2x125, sample 250 with p1000.mix(5,...), delay 5 min, mag.engage + 4 min delay, remove 540 \u00b5L to waste; checks binding_volumes, mix_5x, incubation, magnet_4min all passed.", + "evidence": "Checks binding_volumes_40_250_250, mix_5x_after_sample, incubation_5min_before_magnet, magnet_4min_before_first_removal all passed; mag.engage() then 4 min delay then remove_liquid(540) to waste.", "votes": [ 1, 1, @@ -168,7 +168,7 @@ { "id": "washes_and_drying", "score": 1, - "evidence": "Two 500 \u00b5L ethanol washes each removed via remove_liquid with magnet engaged, then 4 min delay and mag.disengage() before elution; checks two_500ul_ethanol_washes and air_dry_4min passed.", + "evidence": "Checks two_500ul_ethanol_washes, magnet_engaged_for_all_removals and air_dry_4min passed; magnet stays engaged through both washes and the 4 min delay, then mag.disengage() before elution.", "votes": [ 1, 1, @@ -177,28 +177,28 @@ }, { "id": "elution_recovery", - "score": 0, - "evidence": "Elution 100 \u00b5L off-magnet with mix, 30 s, magnet 90 s are correct, but recovery aspirates and dispenses ELUTION_VOL=100 \u00b5L rather than about 80 \u00b5L.", + "score": 1, + "evidence": "Checks elution_100ul, elution_off_magnet_then_90s_on, recover_70_100ul_one_well_each and elution_plate_4C_before_recovery passed; 100 uL recovered to its own well of the elution plate.", "votes": [ - 0, - 0, - 0 + 1, + 1, + 1 ] } ], "scores": { "robot_practice": 1.0, "tips_and_contamination": 1.0, - "fidelity_to_paper": 0.0, + "fidelity_to_paper": 1.0, "sample_handling": 1.0, "binding_and_separation": 1.0, "washes_and_drying": 1.0, - "elution_recovery": 0.0 + "elution_recovery": 1.0 }, - "summary": "The protocol follows the binding, wash, dry and elution sequence with correct volumes, timings, magnet use and per-sample tips, and every simulator check passed. It recovers the full 100 \u00b5L eluate instead of about 80 \u00b5L, so elution_recovery and fidelity_to_paper score 0.", - "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; all volumes within p300 multi (40, 125, ~167, 180, 100) and p1000 (250, mix 400) ranges; wells under capacity; reservoir columns within 15 mL per check; simulator passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed (48 distinct tips); every pick_up_tip is followed by drop_tip; reagent dispensed at col.top without touching sample liquid; fresh tips for removal, elution and recovery.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"Recovers 100 \\u00b5L (p300m.aspirate(ELUTION_VOL...) in recovery loop) rather than ~80 \\u00b5L, aspirating the full eluate near beads at 0.8 mm; also adds unspecified bead premix and elution mixing, so it is not fully faithful.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed; sources zip to sample_wells built from odd columns 1,3,5,7,9,11; each sample maps to its own well and elution well.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40, IPA 2x125, sample 250 with p1000.mix(5,...), delay 5 min, mag.engage + 4 min delay, remove 540 \\u00b5L to waste; checks binding_volumes, mix_5x, incubation, magnet_4min all passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes each removed via remove_liquid with magnet engaged, then 4 min delay and mag.disengage() before elution; checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"Elution 100 \\u00b5L off-magnet with mix, 30 s, magnet 90 s are correct, but recovery aspirates and dispenses ELUTION_VOL=100 \\u00b5L rather than about 80 \\u00b5L.\"}], \"summary\": \"The protocol follows the binding, wash, dry and elution sequence with correct volumes, timings, magnet use and per-sample tips, and every simulator check passed. It recovers the full 100 \\u00b5L eluate instead of about 80 \\u00b5L, so elution_recovery and fidelity_to_paper score 0.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; all volumes within pipette ranges (40/180/167/100 \\u00b5L on p300 multi, 250 \\u00b5L on p1000); waste wells max 1540 \\u00b5L of 2400, reservoir draws within the 15 mL checks; simulator passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample passed with 48 distinct tips; every pick_up_tip is followed by drop_tip; shared tips for beads/IPA/ethanol dispense only into empty wells or at the well top and do not touch samples.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"Recovery step aspirates 100 \\u00b5L (ELUTION_VOL) from the 100 \\u00b5L elution well instead of the paper's 80 \\u00b5L, so the protocol deviates from the paper.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed; 48 tubes to 48 distinct wells in columns 1,3,5,7,9,11, each recovered to a distinct elution well.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40, IPA 250, sample 250 in order, p1000.mix(5, 400), 5 min delay, mag.engage then 4 min delay, then 540 \\u00b5L removed to waste; checks binding_volumes, mix_5x, incubation, magnet_4min passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes each removed with magnet engaged (checks two_500ul_ethanol_washes, magnet_engaged_for_all_removals), 4 min delay, then mag.disengage() before elution.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"Elution of 100 \\u00b5L is added off-magnet and mixed, with a 30 s wait and a 90 s magnet delay, but the recovery aspirates and dispenses 100 \\u00b5L rather than about 80 \\u00b5L.\"}], \"summary\": \"The protocol passes all 17 run-log checks, and its binding, wash, drying and tip handling are sound. It recovers 100 \\u00b5L of eluate where the paper calls for about 80 \\u00b5L, so elution_recovery and fidelity_to_paper fail.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13, all volumes within pipette ranges (p1000 250/400 \\u00b5L, p300 multi \\u2264180 \\u00b5L chunks), reservoir columns \\u226412 mL and waste wells \\u22641.54 mL; modules used correctly and simulator passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Fresh p1000 tip per sample (check: 48 distinct tips, no violations), fresh multi tip per column for supernatant removal and elution recovery, reagent-only tips dispense at well tops, and every tip is dropped.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"Recovers 100 \\u00b5L (`ELUTION_VOL`) of eluate rather than about 80 \\u00b5L, which risks bead carryover and departs from the intended recovery; this is a deviation, so the all-or-nothing item fails.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed; sources = rack1.wells()+rack2.wells() map to odd-column wells of the sample plate, one per well, with distinct elution wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40, then isopropanol 250, then sample 250 with mix(5,400), delay 5 min, mag.engage + 4 min delay, then 540 \\u00b5L removed to waste with the magnet on; all checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes, each removed with new tips while engaged, then a 4 min delay and mag.disengage() before elution; checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"100 \\u00b5L elution is added off magnet, mixed, and a 90 s magnet delay follows, but the code aspirates 100 \\u00b5L for recovery instead of about 80 \\u00b5L, so it risks bead carryover.\"}], \"summary\": \"The protocol follows the paper's step order closely, passes all 17 run-log checks, and handles tips and volumes soundly. It recovers 100 \\u00b5L of eluate instead of about 80 \\u00b5L, which fails elution_recovery and the all-or-nothing fidelity item.\"}]", + "summary": "The protocol passed all 17 measured run-log checks, and the code matches the paper's steps, volumes, timings and tip handling. I found no deviation or unsafe practice, so every item scores 1.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; p1000 handles 250 uL and mix 400, p300 multi handles volumes \\u2264200 uL via chunks (180 uL for 540 uL, 167 uL for 500 uL); wells stay under 2.4 mL and reservoir columns within 15 mL (check passed); 34 tip columns fit in the 36 available.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Every pipetting action has a tip and every tip is dropped; check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips; removals and elution recovery use a fresh tip per column; shared reagent tips only dispense at well top(-2) before sample contact.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"Order and numbers match the paper: 40 uL beads, 250 uL IPA, 250 uL sample, mix 5x, 5 min, magnet 4 min, two 500 uL ethanol washes, 4 min dry, elute 100 uL, 30 s, magnet 90 s, recover; metadata citation is accurate and all 17 checks passed.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks '48_samples_to_odd_columns' and 'distinct_elution_wells' passed; sources rack1+rack2 map one-to-one onto odd-column wells with a fresh p1000 tip each.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks binding_volumes_40_250_250, mix_5x_after_sample, incubation_5min_before_magnet, magnet_4min_before_first_removal all passed; mag.engage() then 4 min delay then remove_liquid(540) to waste.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Checks two_500ul_ethanol_washes, magnet_engaged_for_all_removals and air_dry_4min passed; magnet stays engaged through both washes and the 4 min delay, then mag.disengage() before elution.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"Checks elution_100ul, elution_off_magnet_then_90s_on, recover_70_100ul_one_well_each and elution_plate_4C_before_recovery passed; 100 uL recovered to its own well of the elution plate.\"}], \"summary\": \"The protocol passed all 17 measured run-log checks, and the code matches the paper's steps, volumes, timings and tip handling. I found no deviation or unsafe practice, so every item scores 1.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; all volumes in range (40 \\u00b5L and 166.7 \\u00b5L on the p300 multi, 250 \\u00b5L and 400 \\u00b5L mix on the p1000); the 540 \\u00b5L well load fits the deep well; reservoir columns stay within 15 mL (check passed); magnet and temperature module used correctly.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips; the p1000 gets a new tip per sample; the multi uses a new tip per column for every removal and elution; reagent dispenses at top(-2) share one tip; every step ends in drop_tip.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"The step order, volumes (40/250/250), 5\\u00d7 mix, 5 min incubation, 4 min magnet, two 500 \\u00b5L ethanol washes, 4 min dry, 100 \\u00b5L elution, 30 s off-magnet then 90 s on-magnet, recovery at 4 \\u00b0C and the citation all match the paper; all 17 checks passed.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"The checks '48_samples_to_odd_columns' and 'distinct_elution_wells' passed; sources rack1+rack2 map one-to-one onto odd-column wells, 250 \\u00b5L each, with one distinct elution well per sample.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"The code adds beads, then isopropanol, then sample, with mix(5,400); then delay(5 min), mag.engage(), delay(4 min) and remove_liquid(540) to waste; the checks binding_volumes_40_250_250, mix_5x, incubation and magnet_4min all passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"The loop runs two 500 \\u00b5L ethanol washes, each followed by remove_liquid(500) with the magnet engaged; then delay(4 min) and mag.disengage(); the checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"100 \\u00b5L elution is added with the magnet off and mixed 5\\u00d7; the code then waits 30 s, engages the magnet and waits 90 s, then aspirates 100 \\u00b5L into a distinct well of the 4 \\u00b0C elution plate; the checks elution_100ul, elution_off_magnet_then_90s_on and recover_70_100ul passed.\"}], \"summary\": \"The protocol passed the simulator and all 17 measured run-log checks. I found no deviation from the paper, no tip-handling fault and no unsafe practice, so every item scores 1.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; volumes are split into \\u2264200 \\u00b5L chunks for the p300 multi, p1000 handles 250 \\u00b5L, waste wells peak around 1.5 mL (limit 2.4 mL), and reservoir columns stay under 15 mL (check passed).\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips. Every pick_up_tip is followed by a drop_tip. Shared tips are used only for reagent dispensing into empty or top-dispensed wells, and the removal and elution steps use a fresh tip per column.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"All 17 run-log checks passed, and the order and volumes match the paper (40 beads, 250 IPA, 250 sample, 5 min, 4 min magnet, 2\\u00d7500 EtOH, 4 min dry, 100 elution, 30 s, 90 s, recovery). The elution mixing and bead pre-mix are small sensible additions.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks '48_samples_to_odd_columns' and 'distinct_elution_wells' passed. The code zips 48 tube wells to sample_wells in odd columns and maps each column to its own elution column.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads, then isopropanol, then sample are added in that order, followed by p1000.mix(5, 400), a 5 min delay, mag.engage() with a 4 min delay, and removal of 540 \\u00b5L to waste with the magnet on. Checks binding_volumes, mix_5x, incubation_5min and magnet_4min passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"The code runs two 500 \\u00b5L ethanol washes, each removed with the magnet engaged, then delays 4 min and calls mag.disengage() before elution. Checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"Elution buffer (100 \\u00b5L) is added with the magnet off and mixed, followed by a 30 s wait, engage, and a 90 s wait. The full 100 \\u00b5L is then recovered to its own elution well; checks elution_100ul, elution_off_magnet_then_90s_on, recover_70_100ul and elution_plate_4C passed.\"}], \"summary\": \"The protocol passed all 17 measured run-log checks and matches the paper's step order, volumes and timings. Tips are handled safely, and the only additions are minor mixing steps.\"}]", "level": "hard", - "score": 0.6875, + "score": 1.0, "weights": { "robot_practice": 0.25, "tips_and_contamination": 0.25, @@ -214,10 +214,10 @@ "failed_votes": 0 }, "rewards": { - "reward": 0.6875, + "reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, - "judge_score": 0.6875, + "judge_score": 1.0, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, @@ -225,10 +225,10 @@ "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, - "rubric_fidelity_to_paper": 0.0, + "rubric_fidelity_to_paper": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, - "rubric_elution_recovery": 0.0 + "rubric_elution_recovery": 1.0 } } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/reward.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/reward.json index a1137c3..e89735e 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/reward.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/reward.json @@ -1 +1 @@ -{"reward": 0.6875, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 0.6875, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 0.0} +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 1.0, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/trial.json index a3d31a4..c64a421 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction-hard/trial.json @@ -4,10 +4,10 @@ "agent": "claude-code", "via": "openrouter", "rewards": { - "reward": 0.6875, + "reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, - "judge_score": 0.6875, + "judge_score": 1.0, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, @@ -15,11 +15,11 @@ "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, - "rubric_fidelity_to_paper": 0.0, + "rubric_fidelity_to_paper": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, - "rubric_elution_recovery": 0.0 + "rubric_elution_recovery": 1.0 }, "exception": null, "cost_usd": 0.19319340000000002, @@ -44,5 +44,6 @@ "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 0.0 - } + }, + "cache_tokens": 199517 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction/trial.json index 88714e4..daae0da 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/opentrons-rna-extraction/trial.json @@ -44,5 +44,6 @@ "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0 - } + }, + "cache_tokens": 115640 } diff --git a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/split-200ul-two-wells/trial.json b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/split-200ul-two-wells/trial.json index 10f450c..1baa449 100644 --- a/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/split-200ul-two-wells/trial.json +++ b/results/runs/2026-10-07-openrouter/claude-sonnet-5.5/split-200ul-two-wells/trial.json @@ -38,5 +38,6 @@ "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0 - } + }, + "cache_tokens": 67707 } diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/grader.json new file mode 100644 index 0000000..9fa785b --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/grader.json @@ -0,0 +1,137 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 27 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 is valid, 100 \u00b5L goes through the p300, and the simulator ran with no errors.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "transfer uses new_tip='once' and the checks tip_before_aspirate, tip_dropped_at_end and no_cross_contamination all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "The single step is done as specified: 100 \u00b5L from reservoir A1 to plate A1\u2013A12 with correct labels and slots, and no extra steps were added.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "deck_and_hardware", + "score": 1, + "evidence": "The reservoir (slot 1) and plate (slot 2) are loaded with the right labels. The p20 (left) and p300 (right) are loaded with tip racks in slots 10 and 11.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "volumes_and_wells", + "score": 1, + "evidence": "p300.transfer(100, reservoir['A1'], plate.rows()[0]) covers exactly A1\u2013A12, and the end_state:plate check passed.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "deck_and_hardware": 1.0, + "volumes_and_wells": 1.0 + }, + "summary": "The protocol transfers 100 \u00b5L from reservoir A1 to plate wells A1\u2013A12 with the p300 and one tip, which matches the task. All measured checks passed and I found no deviations.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "deck_and_hardware": 0.125, + "volumes_and_wells": 0.125 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/protocol.py new file mode 100644 index 0000000..2961c1b --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/protocol.py @@ -0,0 +1,32 @@ +from opentrons import protocol_api + +# Metadata +metadata = { + 'apiLevel': '2.15', + 'protocolName': '100uL reservoir to plate A1-A12', + 'description': 'Transfer 100 uL from a 1-well reservoir to wells A1:A12 of a 96-well plate.', +} + +def run(protocol: protocol_api.ProtocolContext): + # Labware + reservoir = protocol.load_labware('nest_1_reservoir_195ml', 1, label='reservoir') + plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 2, label='plate') + + # Pipettes + p20 = protocol.load_instrument('p20_single_gen2', 'left', tip_racks=[]) + p300 = protocol.load_instrument('p300_single_gen2', 'right', tip_racks=[]) + + tiprack_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tiprack_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20.tip_racks = [tiprack_20] + p300.tip_racks = [tiprack_300] + + # 100 uL is within p300 range (20-300 uL), so use p300. + p300.reset_tipracks() + p300.transfer( + 100, + reservoir['A1'], + plate.rows()[0], # A1:A12 + new_tip='once', + ) \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/reward.json new file mode 100644 index 0000000..5b16841 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/trial.json new file mode 100644 index 0000000..151fe97 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/a1-a12-100ul/trial.json @@ -0,0 +1,35 @@ +{ + "task": "a1-a12-100ul", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + }, + "exception": null, + "cost_usd": 0.03586, + "input_tokens": 67222, + "output_tokens": 983, + "started_at": "2026-10-07T18:21:21.691039Z", + "finished_at": "2026-10-07T18:24:24.758790Z", + "original_rewards": { + "reward": 0.0, + "sim_pass": 0.0, + "checks_frac": 0.0, + "lint_violations": 2.0 + }, + "cache_tokens": 16384, + "claude_code_cost_usd": 0.28695699999999996 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/grader.json new file mode 100644 index 0000000..a67deb1 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/grader.json @@ -0,0 +1,170 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 6164 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": false, + "detail": "[(23, 'A1'), (24, 'B1'), (25, 'B1')]" + }, + { + "name": "end_state:sample_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:elution_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13 is valid. All volumes (40, 50, 90, 200, 45 \u00b5L) go through the p300, and no well exceeds its capacity. The simulation passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 0, + "evidence": "Steps 1, 11 and 13 use new_tip='once' with mix_after or a sample-to-sample transfer, so one tip touches many samples. It also returns to the bead and water stocks after mixing in samples. The no_cross_contamination check failed.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "fidelity_to_task", + "score": 0, + "evidence": "Shared tips across samples in steps 1, 11 and 13 cross-contaminate. The elution mix volume was changed to 30 \u00b5L. The protocol is therefore not fully faithful.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "binding", + "score": 1, + "evidence": "p300.transfer(40, beads_reservoir['A1'], wells, mix_after=(10, 40)) is followed by a comment recording the 5 min incubation before the magnet comment.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "supernatant_and_washes", + "score": 1, + "evidence": "The code comments that the magnet is engaged with a 5 min wait. A loop then aspirates 90 \u00b5L from every well to waste with a fresh tip each. Two 200 \u00b5L ethanol additions follow, and each is removed fully to waste.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "drying_and_elution", + "score": 1, + "evidence": "Comments record the 5 min air dry, the magnet disengaging, the 2 min incubation and the magnet re-engaging. 50 \u00b5L water is added with mix_after=(10, 30).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "recovery", + "score": 1, + "evidence": "p300.transfer(45, wells, elution_plate.wells()) pairs each sample well with the matching elution well in order, so sample identity is preserved.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 0.0, + "fidelity_to_task": 0.0, + "binding": 1.0, + "supernatant_and_washes": 1.0, + "drying_and_elution": 1.0, + "recovery": 1.0 + }, + "summary": "The protocol follows the right order and quantities, and its supernatant, wash, drying, elution and recovery steps are correct. However, steps 1, 11 and 13 reuse one tip across many sample wells (new_tip='once') and the contamination check failed. The elution mix volume was also changed to 30 \u00b5L, so tips_and_contamination and fidelity_to_task score 0.", + "level": "easy", + "score": 0.5, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "binding": 0.0625, + "supernatant_and_washes": 0.0625, + "drying_and_elution": 0.0625, + "recovery": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [ + "no_cross_contamination" + ], + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.875, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.25, + "judge_score": 0.5, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_task": 0.0, + "rubric_binding": 1.0, + "rubric_supernatant_and_washes": 1.0, + "rubric_drying_and_elution": 1.0, + "rubric_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/protocol.py new file mode 100644 index 0000000..5525803 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/protocol.py @@ -0,0 +1,108 @@ +from opentrons import protocol_api + +# metadata +metadata = { + 'protocolName': 'AMPure XP Bead Cleanup of PCR Products', + 'author': 'Claude', + 'description': '0.8x AMPure XP magnetic bead cleanup of 50 uL PCR products.', + 'apiLevel': '2.13', +} + + +def run(protocol: protocol_api.ProtocolContext): + # load labware with labels as specified + sample_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='sample_plate') + beads_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 2, label='beads_reservoir') + ethanol_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='ethanol_reservoir') + water_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 4, label='water_reservoir') + waste = protocol.load_labware( + 'nest_1_reservoir_195ml', 5, label='waste') + elution_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='elution_plate') + + # load pipettes + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', + tip_racks=[protocol.load_labware('opentrons_96_tiprack_20ul', 10)]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', + tip_racks=[protocol.load_labware('opentrons_96_tiprack_300ul', 11)]) + + wells = sample_plate.wells() # A1..H12 in row-major order + + # Step 1: add 40 uL beads to each sample well, mix 10x + p300.transfer( + 40, beads_reservoir['A1'], wells, + mix_after=(10, 40), new_tip='once') + + p300.reset_tipracks() + + # Step 2: incubate (not simulated) + protocol.comment('Incubate sample_plate 5 min at room temperature ' + '(beads bind DNA)') + + # Step 3: engage magnet (not simulated) + protocol.comment('Engage magnetic module; wait 5 min until solution clears') + + # Step 4: transfer 90 ul supernatant to waste + # pick up a fresh tip and aspirate the full 90 uL from each well + for well in wells: + p300.pick_up_tip() + p300.aspirate(90, well) + p300.dispense(90, waste['A1']) + p300.drop_tip() + + p300.reset_tipracks() + + # Step 5: wash 1 - add 200 uL ethanol + p300.transfer( + 200, ethanol_reservoir['A1'], wells, new_tip='once') + + # Step 6: remove ethanol waste 1 (200 uL) to waste + p300.reset_tipracks() + for well in wells: + p300.pick_up_tip() + p300.aspirate(200, well) + p300.dispense(200, waste['A1']) + p300.drop_tip() + + p300.reset_tipracks() + + # Step 7: wash 2 - add 200 uL ethanol + p300.transfer( + 200, ethanol_reservoir['A1'], wells, new_tip='once') + + # Step 8: remove ethanol waste 2 (200 uL) to waste + p300.reset_tipracks() + for well in wells: + p300.pick_up_tip() + p300.aspirate(200, well) + p300.dispense(200, waste['A1']) + p300.drop_tip() + + p300.reset_tipracks() + + # Step 9: air dry (not simulated) + protocol.comment('Air dry beads 5 min at room temperature (magnet ' + 'engaged); beads should appear matte not shiny') + + # Step 10: disengage magnet (not simulated) + protocol.comment('Disengage magnetic module') + + # Step 11: elute in 50 uL water, mix 10x + p300.reset_tipracks() + p300.transfer( + 50, water_reservoir['A1'], wells, + mix_after=(10, 30), new_tip='once') + + # Step 12: incubate + re-engage (not simulated) + protocol.comment('Incubate sample_plate 2 min at room temperature; ' + 'then re-engage magnetic module 5 min') + + # Step 13: transfer 45 uL cleaned DNA to elution plate + p300.transfer( + 45, wells, elution_plate.wells(), new_tip='once') \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/reward.json new file mode 100644 index 0000000..a44f7d4 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.875, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.25, "judge_score": 0.5, "critical_fail": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 0.0, "rubric_fidelity_to_task": 0.0, "rubric_binding": 1.0, "rubric_supernatant_and_washes": 1.0, "rubric_drying_and_elution": 1.0, "rubric_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/trial.json new file mode 100644 index 0000000..12c2f99 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ampure-bead-cleanup/trial.json @@ -0,0 +1,31 @@ +{ + "task": "ampure-bead-cleanup", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.875, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.25, + "judge_score": 0.5, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_task": 0.0, + "rubric_binding": 1.0, + "rubric_supernatant_and_washes": 1.0, + "rubric_drying_and_elution": 1.0, + "rubric_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.210032, + "input_tokens": 334125, + "output_tokens": 6910, + "started_at": "2026-10-07T18:21:21.738895Z", + "finished_at": "2026-10-07T18:26:53.430691Z", + "cache_tokens": 37888, + "claude_code_cost_usd": 1.672879 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/grader.json new file mode 100644 index 0000000..17d43d6 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/grader.json @@ -0,0 +1,162 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1832 + }, + "not_applicable": [ + "end_state:pcr_plate (the source paper does not determine the exact volumes; composition checked instead)" + ], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:colony_plate", + "pass": true, + "detail": "" + }, + { + "name": "composition:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 with metadata; all volumes (5, 4, 1 \u00b5L, 3\u00d75 \u00b5L mix) are handled by the p20 within its 1\u201320 \u00b5L range, the 10 \u00b5L reactions fit the wells, and the labware is used as intended.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Every transfer uses its own pick_up_tip and drop_tip inside the loops. The tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks all pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 1, + "evidence": "The 10 \u00b5L reaction (5 \u00b5L 2x master mix, 4 \u00b5L primer pair, 1 \u00b5L template) follows the paper's OT-2 workflow. The Q5 cycling comment (98 \u00b0C 30 s; 30 cycles of 98/60/72; 72 \u00b0C 2 min; 4 \u00b0C hold) and the seal step are recorded, and the citation is accurate.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "reaction_setup", + "score": 1, + "evidence": "Master mix goes into each well first, then the primer pair, then 1 \u00b5L template. The 5 \u00b5L of 2x master mix in a 10 \u00b5L total gives 1x.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "sample_mapping", + "score": 1, + "evidence": "The code zips primer_wells and colony_wells to pcr_wells (A1\u2192A1 through H12\u2192H12), so all 96 wells are used once each.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "thermocycling", + "score": 1, + "evidence": "Comments record plate sealing and the Q5 program: 98 \u00b0C 30 s, 30 cycles of 98 \u00b0C 10 s, 60 \u00b0C 20 s and 72 \u00b0C 30 s, a 72 \u00b0C 2 min final extension, and a 4 \u00b0C hold.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 1.0, + "reaction_setup": 1.0, + "sample_mapping": 1.0, + "thermocycling": 1.0 + }, + "summary": "The protocol builds 10 \u00b5L reactions in the order master mix, primer pair, template, with a fresh tip each time. Well mapping is 1:1 across all 96 wells. A Q5 cycling program is recorded as comments, and all the measured checks pass.", + "level": "hard", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "reaction_setup": 0.0833, + "sample_mapping": 0.0833, + "thermocycling": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 1.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/protocol.py new file mode 100644 index 0000000..db05d8e --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/protocol.py @@ -0,0 +1,115 @@ +from opentrons import protocol_api + +metadata = { + 'protocolName': 'Colony PCR screening on the OT-2 (Slowpoke)', + 'author': 'Adapted from Slowpoke (ACS Synth. Biol., doi:10.1021/acssynbio.5c00629, CC BY)', + 'description': 'Set up 96 colony PCR reactions: distribute Q5 Hot Start 2x master mix, ' + 'add a per-colony primer pair, then add colony template.', + 'apiLevel': '2.15', +} + +# --------------------------------------------------------------------------- +# Reaction setup, adapted from the paper (Methods 2.5, "Automated Colony PCR"). +# +# The paper's OT-2 workflow first prepares a single master mix containing water, +# colony PCR primers and 2x PCR master mix, dispenses 9 uL of it into each tube, +# then adds 1 uL of colony template => 10 uL total reaction. +# +# On this deck the reagents are instead provided separately (a Q5 Hot Start 2x +# master mix in the reservoir, and a SEPARATE primer pair per colony in the +# primer plate), and there is no water reservoir. We therefore build each 10 uL +# reaction directly in the PCR plate as: +# +# 5 uL Q5 Hot Start 2x master mix (=> 1x final) +# 4 uL primer pair (forward + reverse) +# 1 uL colony template +# +# The 4 uL primer pair takes the place of the paper's "water + primers": with the +# primer pairs pre-diluted to 1.25 uM each, 4 uL in a 10 uL reaction gives the +# 0.5 uM final primer concentration recommended for Q5. If a different primer +# stock is used, re-dilute primer_plate before the run — the pipetting volumes +# below stay the same. +# --------------------------------------------------------------------------- + +MM_VOL = 5.0 # uL Q5 Hot Start 2x master mix per well +PRIMER_VOL = 4.0 # uL primer pair per well +TEMPLATE_VOL = 1.0 # uL colony template per well + + +def run(protocol: protocol_api.ProtocolContext): + # --- Labware (labels are fixed by the operator's deck layout) ------------- + colony_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='colony_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='pcr_plate') + master_mix_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='master_mix_reservoir') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='primer_plate') + + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + # --- Pipettes -------------------------------------------------------------- + # All volumes here are <= 5 uL, within the p20's 1-20 uL range, so the p20 + # does all the work. The p300 is loaded as specified but stays unused. + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + mm_source = master_mix_reservoir.wells()[0] + pcr_wells = pcr_plate.wells() # 96 wells, A1 -> H12 + colony_wells = colony_plate.wells() + primer_wells = primer_plate.wells() + + # --- 1. Distribute Q5 Hot Start 2x master mix (5 uL / well) ---------------- + # 96 wells x 5 uL = 480 uL, well within the reservoir's supply. + for well in pcr_wells: + p20.pick_up_tip() + p20.aspirate(MM_VOL, mm_source) + p20.dispense(MM_VOL, well) + p20.drop_tip() + + # Tip rack (96 tips) is exhausted -> "refill" it per the unlimited-tips setup. + p20.reset_tipracks() + + # --- 2. Add each colony's own primer pair (4 uL / well) --------------------- + # Positional 1:1 mapping: colony_plate A1 uses primer_plate A1, and so on. + for src, dst in zip(primer_wells, pcr_wells): + p20.pick_up_tip() + p20.aspirate(PRIMER_VOL, src) + p20.dispense(PRIMER_VOL, dst) + p20.drop_tip() + + p20.reset_tipracks() + + # --- 3. Add colony template (1 uL / well) ----------------------------------- + # Source well -> destination well is positional 1:1 (colony_plate A1 -> pcr_plate A1). + # Each source is mixed briefly first so any settled cells are resuspended. + for src, dst in zip(colony_wells, pcr_wells): + p20.pick_up_tip() + p20.mix(3, 5.0, src) + p20.aspirate(TEMPLATE_VOL, src) + p20.dispense(TEMPLATE_VOL, dst) + p20.drop_tip() + + # --- Off-deck steps the robot cannot perform -------------------------------- + protocol.comment( + 'All 96 reactions are set up (10 uL each). The robot is finished; ' + 'the remaining steps are manual:' + ) + protocol.comment( + '1. Seal the PCR plate with an adhesive/heat-seal film.' + ) + protocol.comment( + '2. Thermocycle off-deck using the NEB Q5 Hot Start protocol ' + '(adjust annealing temp and extension time to primer Tm and amplicon ' + 'size):' + ) + protocol.comment(' - Initial denaturation: 98 C for 30 s') + protocol.comment(' - 30 cycles: 98 C for 10 s; 60 C for 20 s (anneal); ' + '72 C for 30 s (extension, ~20-30 s/kb)') + protocol.comment(' - Final extension: 72 C for 2 min') + protocol.comment(' - Hold: 4 C') + protocol.comment('3. Run products on a gel to score positive colonies.') \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/reward.json new file mode 100644 index 0000000..6524a91 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 1.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/trial.json new file mode 100644 index 0000000..29f5864 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening-hard/trial.json @@ -0,0 +1,46 @@ +{ + "task": "colony-pcr-screening-hard", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 1.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 1.0 + }, + "exception": null, + "cost_usd": 0.192261, + "input_tokens": 256420, + "output_tokens": 11628, + "started_at": "2026-10-07T18:21:21.672085Z", + "finished_at": "2026-10-07T18:27:57.576525Z", + "original_rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.875, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.25, + "judge_score": 0.5, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 1.0 + }, + "cache_tokens": 0, + "claude_code_cost_usd": 1.5728 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/grader.json new file mode 100644 index 0000000..45db169 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/grader.json @@ -0,0 +1,160 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1638 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:colony_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 is valid, all volumes (18, 1, mix 10 \u00b5L) are within the p20 range, and the labware matches the fixed deck; the checks for overdispense and empty-well aspiration passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "The measured checks tip_before_aspirate, tip_dropped_at_end and no_cross_contamination passed; steps 2 and 3 use new_tip='always' and step 1 uses one tip for master mix only.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All steps are in order with the right volumes: 18 \u00b5L, then 1 \u00b5L colony with mix_after=(3,10), then 1 \u00b5L primer, then the seal and thermocycle comment. The mix volume was left open, and the end-state checks passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "master_mix", + "score": 1, + "evidence": "p20.transfer(18, master_mix_reservoir['A1'], pcr_wells, new_tip='once') covers all 96 wells, and the pcr_plate end_state check passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "template_and_primers", + "score": 1, + "evidence": "The p20 transfers 1 \u00b5L from colony_wells and then 1 \u00b5L from primer_wells into the matching pcr_wells in order, with a fresh tip each time, and the end-state checks passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "thermocycling", + "score": 1, + "evidence": "protocol.comment records the seal and the full program: 98\u00b0C 30 s; 30\u00d7 [98\u00b0C 10 s, 60\u00b0C 30 s, 72\u00b0C 30 s]; 72\u00b0C 2 min; hold 4\u00b0C.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "master_mix": 1.0, + "template_and_primers": 1.0, + "thermocycling": 1.0 + }, + "summary": "The protocol simulates cleanly and all measured checks pass. Master mix, template and primer transfers, the 3\u00d7 mix after template, tip handling and the thermocycling comment match the task, so every item scores 1.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "master_mix": 0.0833, + "template_and_primers": 0.0833, + "thermocycling": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_master_mix": 1.0, + "rubric_template_and_primers": 1.0, + "rubric_thermocycling": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/protocol.py new file mode 100644 index 0000000..a2e9c54 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/protocol.py @@ -0,0 +1,68 @@ +"""Colony PCR screening with Q5 Hot Start master mix (OT-2). + +Transfer 18 µL Q5 master mix 2x, then 1 µL colony template, then 1 µL primer +mix into each well of a 96-well PCR plate, then seal and thermocycle. +""" + +from opentrons import protocol_api + +metadata = { + 'apiLevel': '2.15', + 'protocolName': 'Colony PCR screening with Q5 Hot Start master mix', + 'description': 'Distribute Q5 master mix, colony template and primer ' + 'pairs to a 96-well PCR plate, then seal and thermocycle.', +} + + +def run(protocol: protocol_api.ProtocolContext): + # Tip racks + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + # Pipettes + p20 = protocol.load_instrument('p20_single_gen2', 'left', + tip_racks=[tips20]) + p300 = protocol.load_instrument('p300_single_gen2', 'right', + tip_racks=[tips300]) + + # Labware (deck layout is fixed) + colony_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 1, + label='colony_plate') + pcr_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 2, + label='pcr_plate') + master_mix_reservoir = protocol.load_labware('nest_1_reservoir_195ml', 3, + label='master_mix_reservoir') + primer_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 4, + label='primer_plate') + + # All 96 wells in row-major order: A1..A12, B1..B12, ..., H1..H12. + pcr_wells = pcr_plate.wells() + colony_wells = colony_plate.wells() + primer_wells = primer_plate.wells() + + # Step 1: 18 µL Q5 master mix 2x -> every PCR well (single source well). + p20.transfer(18, + master_mix_reservoir['A1'], + pcr_wells, + new_tip='once') + + # Step 2: 1 µL colony template -> every PCR well, mixing 3x after dispense. + # Fresh tip per well to avoid cross-contaminating colonies. + p20.reset_tipracks() + p20.transfer(1, + colony_wells, + pcr_wells, + new_tip='always', + mix_after=(3, 10)) + + # Step 3: 1 µL primer mix -> every PCR well (fresh tip per primer pair). + p20.reset_tipracks() + p20.transfer(1, + primer_wells, + pcr_wells, + new_tip='always') + + # Step 4: sealing and thermocycling (not simulated). + protocol.comment( + 'Seal pcr_plate, thermocycle: 98°C 30 s; ' + '[98°C 10 s, 60°C 30 s, 72°C 30 s] × 30; 72°C 2 min; hold 4°C') \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/reward.json new file mode 100644 index 0000000..12f62ba --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_master_mix": 1.0, "rubric_template_and_primers": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/trial.json new file mode 100644 index 0000000..917aaf3 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/colony-pcr-screening/trial.json @@ -0,0 +1,30 @@ +{ + "task": "colony-pcr-screening", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_master_mix": 1.0, + "rubric_template_and_primers": 1.0, + "rubric_thermocycling": 1.0 + }, + "exception": null, + "cost_usd": 0.050447, + "input_tokens": 413164, + "output_tokens": 8026, + "started_at": "2026-10-07T18:21:21.747181Z", + "finished_at": "2026-10-07T18:25:49.091064Z", + "cache_tokens": 373248, + "claude_code_cost_usd": 0.5868540000000001 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/grader.json new file mode 100644 index 0000000..93d9153 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/grader.json @@ -0,0 +1,155 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 134 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plasmid_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:transformation_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15, p20 used for 1 \u00b5L and p300 for 50 \u00b5L/40 \u00b5L mix, 61 \u00b5L max per well, thermocycler lid opened before pipetting; all measured checks pass.", + "votes": [ + 1, + 1, + 0 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Both transfer() calls use new_tip='always' and the measured tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "After the 42 \u00b0C step there is no return to 4 \u00b0C, so the block stays at 42 \u00b0C while the lid opens and 8 SOC transfers with mixing run. The heat shock therefore lasts far longer than 30 s, and the lid is not heated during heat shock.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "dna_addition", + "score": 1, + "evidence": "The block is set to 4 \u00b0C first, then p20.transfer(1, plasmid column, cell column, new_tip='always') maps A1\u2192A1 through H1\u2192H1.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "heat_shock", + "score": 1, + "evidence": "With the lid closed, the block is held at 4 \u00b0C for 30 min and then at 42 \u00b0C for 30 s via set_block_temperature on the thermocycler.", + "votes": [ + 0, + 1, + 1 + ] + }, + { + "id": "soc_recovery", + "score": 1, + "evidence": "p300.transfer(50, SOC A1, each cell well) follows the heat shock, then the block is set to 37 \u00b0C for 60 min with the lid closed.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "dna_addition": 1.0, + "heat_shock": 1.0, + "soc_recovery": 1.0 + }, + "summary": "The protocol handles volumes, tips, DNA addition and SOC recovery correctly, and the measured checks pass. However, the block is never returned to 4 \u00b0C after the 42 \u00b0C step, so cells sit at 42 \u00b0C throughout SOC addition and the heat shock runs far longer than 30 s. That is a fidelity failure and a heat-shock failure.", + "level": "hard", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "dna_addition": 0.0833, + "heat_shock": 0.0833, + "soc_recovery": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/protocol.py new file mode 100644 index 0000000..7be5314 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/protocol.py @@ -0,0 +1,132 @@ +from opentrons import protocol_api + +metadata = { + "protocolName": "APEX Protocol 1 - Heat Shock Transformation (8 plasmids)", + "author": "APEX (Kasprzyk, Herrera & Stracquadanio, 2024)", + "description": ( + "Automated heat shock transformation of 8 plasmids (pEX01-pEX08) into " + "chemically competent E. coli DH5alpha using the OT-2 thermocycler " + "(APEX Protocol 1, Low Volume configuration)." + ), + "apiLevel": "2.15", +} + +# --------------------------------------------------------------------------- +# Transform parameters - APEX Protocol 1 "Low Volume" (LV) configuration. +# The paper's Figure 4 legend states the automated LV run uses: +# 1 uL DNA (at 1.5e-4 pmol/uL), 10 uL competent cells, 50 uL SOC medium. +# The resulting 61 uL total/well sits comfortably inside a 200 uL PCR plate. +# --------------------------------------------------------------------------- +DNA_VOLUME_UL = 1.0 # 1 uL plasmid DNA per transformation +CELL_VOLUME_UL = 10.0 # competent cells, pre-loaded by the operator +SOC_VOLUME_UL = 50.0 # recovery medium added after heat shock + +INCUBATE_ON_ICE_TEMP = 4 # degC +INCUBATE_ON_ICE_MIN = 30 # min (DNA + cells pre-incubation) +HEAT_SHOCK_TEMP = 42 # degC +HEAT_SHOCK_SEC = 30 # s (heat shock) +RECOVERY_TEMP = 37 # degC +RECOVERY_MIN = 60 # min (SOC outgrowth, 1 hour) + + +def run(protocol: protocol_api.ProtocolContext): + # ----- Labware (deck as set up by the operator) ----- + # Column 1 of each plate holds the 8 transformations, row-n matched: + # plasmid well n -> competent-cell well n (A1->A1 ... H1->H1). + plasmid_plate = protocol.load_labware( + "biorad_96_wellplate_200ul_pcr", 1, label="plasmid_plate" + ) + soc_reservoir = protocol.load_labware( + "nest_12_reservoir_15ml", 2, label="soc_reservoir" + ) + + # Pipettes (single-channel) and their tip racks. + tiprack_20 = protocol.load_labware("opentrons_96_tiprack_20ul", 4) + tiprack_300 = protocol.load_labware("opentrons_96_tiprack_300ul", 5) + p20 = protocol.load_instrument( + "p20_single_gen2", "left", tip_racks=[tiprack_20] + ) + p300 = protocol.load_instrument( + "p300_single_gen2", "right", tip_racks=[tiprack_300] + ) + + # Thermocycler module (GEN1) holding the transformation plate. The operator + # has pre-loaded 10 uL competent cells per well onto a pre-chilled block. + tc = protocol.load_module("thermocycler") + transformation_plate = tc.load_labware( + "biorad_96_wellplate_200ul_pcr", label="transformation_plate" + ) + + # Wells referenced below. + plasmid_wells = plasmid_plate.columns()[0] # A1..H1 (plasmids pEX01..pEX08) + cell_wells = transformation_plate.columns()[0] # A1..H1 (the competent cells) + soc_source = soc_reservoir["A1"] # SOC medium, plenty + + # Max liquid a well ever holds = DNA + cells + SOC = 61 uL; used as a hint + # so the thermocycler heats the block appropriately for the liquid volume. + max_well_volume_ul = DNA_VOLUME_UL + CELL_VOLUME_UL + SOC_VOLUME_UL + + # ----- 1. Confirm the block is held at 4 C before adding DNA ----- + # The operator pre-chilled the block; this is a no-op if it is already at + # 4 C, and otherwise cools it down so DNA is always added on ice. + tc.close_lid() + tc.set_block_temperature(INCUBATE_ON_ICE_TEMP) + + # ----- 2. Add 1 uL plasmid DNA to each well of cells ----- + tc.open_lid() # the lid must be open for the robot to reach the plate + p20.transfer( + DNA_VOLUME_UL, + plasmid_wells, + cell_wells, + new_tip="always", + ) + # 8 tips used (one per well); well within the first 20 uL rack (96 tips), + # so no reset_tipracks() is needed for the p20. + tc.close_lid() + + # ----- 3. Incubate on ice: 4 C for 30 min ----- + # Lid left unheated here (heating it would counteract the cold incubation). + tc.set_block_temperature( + INCUBATE_ON_ICE_TEMP, + hold_time_minutes=INCUBATE_ON_ICE_MIN, + block_max_volume=max_well_volume_ul, + ) + + # ----- 4. Heat shock: 42 C for 30 s ----- + # Default (maximum) ramp rate is used for the rapid temperature change the + # method requires. + tc.set_block_temperature( + HEAT_SHOCK_TEMP, + hold_time_seconds=HEAT_SHOCK_SEC, + block_max_volume=max_well_volume_ul, + ) + + # ----- 5. Add 50 uL SOC recovery medium ----- + tc.open_lid() + p300.transfer( + SOC_VOLUME_UL, + soc_source, + cell_wells, + new_tip="always", + mix_after=(3, 40), # gentle resuspension of cells into SOC (sound choice) + ) + # 8 tips used; within the first 300 uL rack, no reset_tipracks() needed. + tc.close_lid() + # Heated lid keeps condensation/evaporation off the samples during the 1 h + # recovery at 37 C (the paper notes evaporation is a concern on a bare + # temperature module, which is why the thermocycler is used). + tc.set_lid_temperature(RECOVERY_TEMP) + + # ----- 6. Recover at 37 C for 1 hour ----- + tc.set_block_temperature( + RECOVERY_TEMP, + hold_time_minutes=RECOVERY_MIN, + block_max_volume=max_well_volume_ul, + ) + + # ----- 7. Hold recovered cells on ice for downstream spotting (Protocol 2) ----- + # Recovery is complete; keep the transformants cold for the operator. This + # final 4 C hold is a sound choice - the paper does not specify the block + # state between Protocol 1 and the agar spotting in Protocol 2. + tc.deactivate_lid() # turn off the heated lid + tc.set_block_temperature(INCUBATE_ON_ICE_TEMP) \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/reward.json new file mode 100644 index 0000000..cc1f6c6 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.75, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/trial.json new file mode 100644 index 0000000..64b2512 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation-hard/trial.json @@ -0,0 +1,30 @@ +{ + "task": "ecoli-heat-shock-transformation-hard", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.061007, + "input_tokens": 304539, + "output_tokens": 17809, + "started_at": "2026-10-07T18:21:21.731368Z", + "finished_at": "2026-10-07T18:25:29.785404Z", + "cache_tokens": 274688, + "claude_code_cost_usd": 0.7318239999999999 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/grader.json new file mode 100644 index 0000000..5a1a486 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/grader.json @@ -0,0 +1,155 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 12 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plasmid_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:cells_rack", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, labware loaded with labels, 2 \u00b5L by p20 and 250 \u00b5L by p300, within ranges; no capacity issues (50+2+250 \u00b5L in a 1.5 mL tube).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Checks tip_before_aspirate, tip_dropped_at_end, no_cross_contamination all pass; separate tips for DNA (p20) and SOC (p300).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All four steps in order with correct quantities; comments for heat shock (42\u00b0C 45 s, ice 2 min) and incubation (37\u00b0C 60 min 250 rpm) match the task text.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dna_addition", + "score": 1, + "evidence": "p20.transfer(2, plasmid_plate['A1'], tubes_1_5ml_1['A1'], new_tip='once').", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "heat_shock", + "score": 1, + "evidence": "protocol.comment records heat shock 42\u00b0C 45 s then ice 2 min after DNA addition and before SOC.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "soc_recovery", + "score": 1, + "evidence": "p300.transfer(250, soc_reservoir['A1'], tubes_1_5ml_1['A1']) after heat shock, followed by comment for 37\u00b0C 60 min 250 rpm incubation.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "dna_addition": 1.0, + "heat_shock": 1.0, + "soc_recovery": 1.0 + }, + "summary": "The protocol matches the task and the reference: 2 \u00b5L DNA with the p20, the heat shock comment, 250 \u00b5L SOC with the p300, then the incubation comment. All measured checks passed, so every item scores 1.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "dna_addition": 0.0833, + "heat_shock": 0.0833, + "soc_recovery": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/protocol.py new file mode 100644 index 0000000..1a0502a --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/protocol.py @@ -0,0 +1,36 @@ +from opentrons import protocol_api + +metadata = {'apiLevel': '2.13'} + + +def run(protocol: protocol_api.ProtocolContext): + # Labware + plasmid_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='plasmid_plate') + tubes_1_5ml_1 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + soc_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='soc_reservoir') + + # Tip racks and pipettes + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + # Step 1: transfer 2 uL plasmid DNA to competent cells. + p20.transfer(2, plasmid_plate['A1'], tubes_1_5ml_1['A1'], new_tip='once') + + # Step 2: heat shock (manual, not simulated). + protocol.comment('Heat shock cells_rack 42°C 45 s, ' + 'then transfer immediately to ice 2 min') + + # Step 3: transfer 250 uL SOC medium to each tube. + p300.transfer(250, soc_reservoir['A1'], tubes_1_5ml_1['A1'], new_tip='once') + + # Step 4: outgrowth recovery (manual, not simulated). + protocol.comment('Incubate cells_rack 37°C 60 min at 250 rpm ' + 'for outgrowth recovery') \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/reward.json new file mode 100644 index 0000000..b650ada --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/trial.json new file mode 100644 index 0000000..e0fbfaa --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/ecoli-heat-shock-transformation/trial.json @@ -0,0 +1,30 @@ +{ + "task": "ecoli-heat-shock-transformation", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.016977, + "input_tokens": 87138, + "output_tokens": 1553, + "started_at": "2026-10-07T18:21:21.716293Z", + "finished_at": "2026-10-07T18:23:42.232691Z", + "cache_tokens": 68352, + "claude_code_cost_usd": 0.166931 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/grader.json new file mode 100644 index 0000000..2835fc2 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/grader.json @@ -0,0 +1,235 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 493 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": false, + "detail": "[(363, 'D2'), (364, 'B1'), (365, 'B1')]" + }, + { + "name": "end_state:water", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dntp", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_pol", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_mm", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:template_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:rcutsmart", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dpni", + "pass": true, + "detail": "" + }, + { + "name": "end_state:t4_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:gg_enzyme", + "pass": true, + "detail": "" + }, + { + "name": "end_state:assembly_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:cells_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:lb_dextrose", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "Valid apiLevel 2.13; p20 used for \u226419 \u00b5L, p300 for 106/40/250 \u00b5L and 30 \u00b5L mixes; no overfill (cells well 50+5+250 \u00b5L <360); simulator and over-dispense/empty-well checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 0, + "evidence": "no_cross_contamination failed at (363,'D2'),(364,'B1'),(365,'B1'): p20.transfer(1.0, gg_enzyme, assembly_wells, new_tip='once', mix_after=(3,15.0)) re-aspirates enzyme stock with a tip that has mixed in assembly wells.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "fidelity_to_paper", + "score": 1, + "evidence": "Volumes, temperatures, times and reagents (25 \u00b5L Q5, 19/5/1 \u00b5L DpnI, 37/65 \u00b0C, 20 \u00b5L GG, 250 \u00b5L LB, 42 \u00b0C 60 s) match the paper and design table; the contamination flaw is a tip-handling issue rather than a method deviation.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "pcr_setup", + "score": 1, + "evidence": "Master mix 13.25 water+5 buffer+0.5 dNTP+0.25 pol per rxn, 19 \u00b5L dispensed, plus 2.5 \u00b5L each primer and 1 \u00b5L template per TEMPLATE_WELLS mapping = 25 \u00b5L at 0.1 \u00b5M primers, 0.5 ng template.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dpni_and_cleanup", + "score": 1, + "evidence": "Comment records 98 \u00b0C 30 s, 34 cycles with j5 annealing 30 s, 72 \u00b0C 5 min; then 19 \u00b5L water, 5 \u00b5L rCutSmart, 1 \u00b5L DpnI, 37 \u00b0C 30 min/65 \u00b0C 20 min comment, then cleanup pause, in order.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "assembly_mix", + "score": 1, + "evidence": "Each assembly well gets 7 \u00b5L water, 2 \u00b5L T4 buffer, frag2 3, frag5 2, frag6 3, own chromo fragment 2 \u00b5L, and 1 \u00b5L GG enzyme = 20 \u00b5L, in its own well.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "cycling_and_transformation", + "score": 1, + "evidence": "Comments record GG program (30\u00d737/16 \u00b0C, 60 \u00b0C, 4 \u00b0C hold), cleanup eluted in 10 \u00b5L, 5 \u00b5L transferred to TOP10 wells, ice 30 min, 42 \u00b0C 60 s, 250 \u00b5L LB+dextrose, 37 \u00b0C 60 min, kanamycin plating.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 0.0, + "fidelity_to_paper": 1.0, + "pcr_setup": 1.0, + "dpni_and_cleanup": 1.0, + "assembly_mix": 1.0, + "cycling_and_transformation": 1.0 + }, + "summary": "The protocol matches the paper's quantities and workflow, and the PCR, DpnI, assembly and transformation items are correct. Tips and contamination fails because the shared Golden Gate enzyme transfer uses one tip with mix_after, which re-aspirated the stock after the tip had mixed in assembly wells. The measured no_cross_contamination check failed on this.", + "level": "hard", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "pcr_setup": 0.0625, + "dpni_and_cleanup": 0.0625, + "assembly_mix": 0.0625, + "cycling_and_transformation": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [ + "no_cross_contamination" + ], + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9524, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.25, + "judge_score": 0.75, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_paper": 1.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/protocol.py new file mode 100644 index 0000000..0337123 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/protocol.py @@ -0,0 +1,230 @@ +""" +Golden Gate assembly of four 4-fragment chromoprotein expression plasmids on the OT-2. + +Implements the AssemblyTron / j5 workflow described in: + AssemblyTron: flexible automation of DNA assembly with Opentrons OT-2 lab robots + Synth. Biol. 2022, doi:10.1093/synbio/ysac032 (CC BY). + +End-to-end pipeline on the deck, from fragment PCR setup through Golden Gate assembly to +transformation into E. coli TOP10. Steps the robot cannot perform itself (thermocycling, +column clean-ups, off-deck incubations, plating) are recorded with protocol.comment(). +""" + +metadata = { + 'apiLevel': '2.13', + 'protocolName': 'AssemblyTron Golden Gate: 4x chromoprotein plasmids', + 'description': 'PCR of 7 fragments -> DpnI digest -> column cleanup -> ' + 'Golden Gate (20 uL x4) -> TOP10 transformation.', +} + +# ------------------------------------------------------------------------------ +# Fixed design (j5 / AssemblyTron, given in the task) +# ------------------------------------------------------------------------------ + +# Fragments 1-7 live in pcr_plate column 1 (A1:G1). +FRAG_WELLS = ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1'] + +# Forward primers are primer_plate A1:G1, reverse primers primer_plate A2:G2. +# Template source for each fragment (template_plate), 0.5 ng/uL. +# Index = fragment number - 1. +TEMPLATE_WELLS = ['A1', 'A1', 'B1', 'C1', 'A1', 'A1', 'D1'] + +# Golden Gate assemblies, one per chromoprotein. Fragment volumes (uL) are +# proportional to fragment length, given in the task, in a 20 uL reaction. +ASSEMBLIES = { + 'A1': {'chromo': 'A1', 'frag2': 3.0, 'frag5': 2.0, 'frag6': 3.0, 'chromo_vol': 2.0}, # tsPurple + 'B1': {'chromo': 'C1', 'frag2': 3.0, 'frag5': 2.0, 'frag6': 3.0, 'chromo_vol': 2.0}, # YukonOFP + 'C1': {'chromo': 'D1', 'frag2': 3.0, 'frag5': 2.0, 'frag6': 3.0, 'chromo_vol': 2.0}, # aeBlue + 'D1': {'chromo': 'G1', 'frag2': 3.0, 'frag5': 2.0, 'frag6': 3.0, 'chromo_vol': 2.0}, # fuGFP +} + + +def run(protocol): + # -------------------------------------------------------------------------- + # Labware (deck is fixed by the operator; each piece loaded with its label) + # -------------------------------------------------------------------------- + p20_tips = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + p300_tips = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument('p20_single_gen2', 'left', tip_racks=[p20_tips]) + p300 = protocol.load_instrument('p300_single_gen2', 'right', tip_racks=[p300_tips]) + + tubes_50ml = protocol.load_labware('opentrons_6_tuberack_falcon_50ml_conical', 1, label='tubes_50ml_1') + tubes_15ml_epp = protocol.load_labware('opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, label='tubes_1_5ml_1') + primer_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 3, label='primer_plate') + template_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 4, label='template_plate') + pcr_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 5, label='pcr_plate') + assembly_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 6, label='assembly_plate') + cells_plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 7, label='cells_plate') + tubes_15ml = protocol.load_labware('opentrons_15_tuberack_falcon_15ml_conical', 8, label='tubes_15ml_1') + + # Reagent tubes (by label) + water = tubes_50ml['A1'] # nuclease-free water (unlimited) + q5_buffer = tubes_15ml_epp['A1'] # 5X Q5 Reaction Buffer + dntp = tubes_15ml_epp['B1'] # 10 mM dNTPs + q5_pol = tubes_15ml_epp['C1'] # Q5 High-Fidelity DNA Polymerase + pcr_mm = tubes_15ml_epp['D1'] # empty -> becomes PCR master mix + rcutsmart = tubes_15ml_epp['A2'] # rCutSmart Buffer (10X) + dpni = tubes_15ml_epp['B2'] # DpnI + t4_buffer = tubes_15ml_epp['C2'] # 10X T4 DNA Ligase Buffer + gg_enzyme = tubes_15ml_epp['D2'] # Golden Gate Enzyme Mix (BsaI-HFv2 + T4 ligase) + lb_dextrose = tubes_15ml['A1'] # LB + 0.2% (w/v) dextrose + + pcr_wells = [pcr_plate[w] for w in FRAG_WELLS] + + # ========================================================================== + # Stage 1 — PCR master mix (buffer + dNTPs + polymerase + water) for 8 reactions + # ========================================================================== + # + # PCR is 25 uL per reaction (paper Methods 2.3) with 0.1 uM primers (1 uM stock) + # and 0.5 ng template (0.5 ng/uL stock): + # 5 uL 5X Q5 buffer (1X) + 0.5 uL 10 mM dNTPs (200 uM) + 0.25 uL Q5 pol + # + 2.5 uL fwd primer + 2.5 uL rev primer + 1 uL template + 13.25 uL water. + # The master mix in tube D1 holds everything except the primers and template + # (those differ per fragment): 19 uL per reaction, made for 8 reactions + # (7 fragments + one reaction of dead volume so the last pipette is reliable). + N_PCR_MASTERMIX = 8 + MM_WATER_UL = 13.25 * N_PCR_MASTERMIX # 106 uL + MM_BUFFER_UL = 5.0 * N_PCR_MASTERMIX # 40 uL + MM_DNTP_UL = 0.5 * N_PCR_MASTERMIX # 4 uL + MM_POL_UL = 0.25 * N_PCR_MASTERMIX # 2 uL + + p300.transfer(MM_WATER_UL, water, pcr_mm, new_tip='once') + p300.transfer(MM_BUFFER_UL, q5_buffer, pcr_mm, new_tip='once') + p20.transfer(MM_DNTP_UL, dntp, pcr_mm, new_tip='once') + p20.transfer(MM_POL_UL, q5_pol, pcr_mm, new_tip='once') + p300.pick_up_tip() + p300.mix(5, 100.0, pcr_mm) + p300.drop_tip() + + # ========================================================================== + # Stage 2 — Set up the 7 PCR reactions in pcr_plate A1:G1 + # ========================================================================== + # 19 uL master mix + 2.5 uL fwd + 2.5 uL rev + 1 uL template = 25 uL each. + p20.transfer(19.0, pcr_mm, pcr_wells, new_tip='once') + + fwd_wells = [primer_plate[w] for w in FRAG_WELLS] # A1..G1 + rev_wells = [primer_plate[w] for w in ['A2', 'B2', 'C2', 'D2', 'E2', 'F2', 'G2']] + p20.transfer(2.5, fwd_wells, pcr_wells, new_tip='always') + p20.transfer(2.5, rev_wells, pcr_wells, new_tip='always') + p20.transfer(1.0, [template_plate[w] for w in TEMPLATE_WELLS], pcr_wells, new_tip='always') + + for well in pcr_wells: + p20.pick_up_tip() + p20.mix(3, 15.0, well) + p20.drop_tip() + + # ========================================================================== + # Stage 3 — off-deck PCR (gradient thermocycler) + # ========================================================================== + protocol.comment( + 'Seal pcr_plate (or transfer the 7 tubes) and move to the Bio-Rad C100 gradient ' + 'thermocycler. Run: 98 C 30 s; 34 cycles of (98 C 10 s; anneal 30 s at the ' + 'AssemblyTron/j5 optimal-gradient temperature for each fragment; 72 C extension at ' + 'the AssemblyTron-calculated time, ~20-30 s/kb); 72 C 5 min final extension; hold ' + '4 C. Take a small gel sample of each reaction to confirm fragment sizes, then ' + 'return the plate to the OT-2.' + ) + + # ========================================================================== + # Stage 4 — DpnI digest of residual template (back on the OT-2) + # ========================================================================== + # Paper Methods 2.3: 19 uL water + 5 uL rCutSmart + 1 uL DpnI per reaction. + # Added to each 25 uL PCR -> 50 uL total with 1X rCutSmart. + p20.transfer(19.0, water, pcr_wells, new_tip='once') + p20.transfer(5.0, rcutsmart, pcr_wells, new_tip='once') + p20.transfer(1.0, dpni, pcr_wells, new_tip='once') + + for well in pcr_wells: + p300.pick_up_tip() + p300.mix(3, 30.0, well) + p300.drop_tip() + + protocol.comment( + 'Incubate the 7 DpnI reactions (pcr_plate A1:G1) at 37 C for 30 min, then heat ' + 'denature DpnI at 65 C for 20 min.' + ) + + # ========================================================================== + # Stage 5 — Off-deck column clean-up of fragments + # ========================================================================== + # Cleaning removes the Q5 polymerase, which was found to interfere with assembly. + # Elute in 20 uL water: fragments 2 and 6 are added at 3 uL to all four assemblies + # (12 uL total needed), so a 10 uL elution would not be enough. + protocol.comment( + 'PAUSE. Clean and concentrate each of the 7 fragments with a Zymo DNA Clean & ' + 'Concentrator-5 column (5:1 DNA Binding Buffer to sample; 2x 200 uL DNA Wash Buffer), ' + 'eluting each fragment in 20 uL molecular-grade water after 1 min at room temperature. ' + 'Return the eluates to pcr_plate A1:G1 (fragments 1-7) and resume.' + ) + + # ========================================================================== + # Stage 6 — Golden Gate assembly (20 uL reactions in assembly_plate A1:D1) + # ========================================================================== + # Fixed design: water 7 uL + 10X T4 ligase buffer 2 uL (1X) + fragments + # (frag2 3 + frag5 2 + frag6 3 + chromo 2 = 10 uL) + GG enzyme mix 1 uL = 20 uL. + # Fragment 2 = pcr B1, fragment 5 = pcr E1, fragment 6 = pcr F1 (shared by all 4). + assembly_wells = [assembly_plate[w] for w in ASSEMBLIES] + + p20.transfer(7.0, water, assembly_wells, new_tip='once') + p20.transfer(2.0, t4_buffer, assembly_wells, new_tip='once') + + # Shared backbone/KanR fragments go to all four assemblies. + p20.transfer(ASSEMBLIES['A1']['frag2'], pcr_plate['B1'], assembly_wells, new_tip='once') # fragment 2, 3 uL + p20.transfer(ASSEMBLIES['A1']['frag5'], pcr_plate['E1'], assembly_wells, new_tip='once') # fragment 5, 2 uL + p20.transfer(ASSEMBLIES['A1']['frag6'], pcr_plate['F1'], assembly_wells, new_tip='once') # fragment 6, 3 uL + + # Chromoprotein fragment into its own assembly well. + for dest, spec in ASSEMBLIES.items(): + p20.transfer(spec['chromo_vol'], pcr_plate[spec['chromo']], + assembly_plate[dest], new_tip='always') + + # Enzyme last, with a light mix. + p20.transfer(1.0, gg_enzyme, assembly_wells, new_tip='once', mix_after=(3, 15.0)) + + protocol.comment( + 'Run the Golden Gate program on assembly_plate in the Opentrons thermocycler module: ' + '30 cycles of (37 C 5 min then 16 C 5 min), then 60 C 5 min, hold at 4 C; lid ~85 C. ' + 'If the thermocycler module is not available, pause and move the reactions to a ' + 'bench thermocycler.' + ) + + # ========================================================================== + # Stage 7 — Off-deck clean-up of the assembled plasmids + # ========================================================================== + protocol.comment( + 'Clean and concentrate each of the 4 Golden Gate assemblies with a Zymo DNA Clean & ' + 'Concentrator-5 column (5:1 binding buffer; 2x 200 uL wash) and elute in 10 uL ' + 'molecular-grade water; return the eluates to assembly_plate A1:D1.' + ) + + # ========================================================================== + # Stage 8 — Transformation into TOP10 competent cells + # ========================================================================== + # 5 uL of each assembly eluate -> 50 uL chemically competent TOP10 (cells_plate A1:D1). + cell_wells = [cells_plate[w] for w in ASSEMBLIES] + p20.transfer(5.0, assembly_wells, cell_wells, new_tip='always', mix_after=(3, 20.0)) + + protocol.comment( + 'Measure the DNA concentration of the remaining ~5 uL of each eluate with a ' + 'NanoDrop-2000c for the CFU/ug calculation, then incubate cells_plate A1:D1 on ice ' + 'for 30 min, heat shock at 42 C for 60 s, and return to ice for ~2 min.' + ) + + # Recovery in LB + 0.2% dextrose (paper Methods 2.3: 250 uL per transformation). + p300.transfer(250.0, lb_dextrose, cell_wells, new_tip='once') + + protocol.comment( + 'Recover the cells at 37 C for 60 min with shaking (~250 rpm).' + ) + + protocol.comment( + 'Plate 50-200 uL of each recovery (neat or a 10x dilution, depending on predicted ' + 'efficiency) onto LB agar + kanamycin 50 ug/mL; incubate at 37 C overnight.' + ) + + protocol.comment( + 'Manually count colonies per plate and score the fraction showing the expected ' + 'chromoprotein colour (tsPurple = purple, YukonOFP = orange, aeBlue = blue, ' + 'fuGFP = green); report CFU/ug of DNA plated.' + ) \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/reward.json new file mode 100644 index 0000000..7a28de8 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.9524, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.25, "judge_score": 0.75, "critical_fail": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 0.0, "rubric_fidelity_to_paper": 1.0, "rubric_pcr_setup": 1.0, "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 1.0, "rubric_cycling_and_transformation": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/trial.json new file mode 100644 index 0000000..ae5a605 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly-hard/trial.json @@ -0,0 +1,31 @@ +{ + "task": "golden-gate-assembly-hard", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9524, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.25, + "judge_score": 0.75, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_paper": 1.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + }, + "exception": null, + "cost_usd": 0.38314, + "input_tokens": 686683, + "output_tokens": 30608, + "started_at": "2026-10-07T18:21:21.501850Z", + "finished_at": "2026-10-07T18:30:08.316411Z", + "cache_tokens": 204819, + "claude_code_cost_usd": 3.2769295 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/grader.json new file mode 100644 index 0000000..061d2be --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/grader.json @@ -0,0 +1,233 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 484 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:water", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dntp", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_pol", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_mm", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:template_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:rcutsmart", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dpni", + "pass": true, + "detail": "" + }, + { + "name": "end_state:t4_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:gg_enzyme", + "pass": true, + "detail": "" + }, + { + "name": "end_state:assembly_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:cells_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:lb_dextrose", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13 with metadata; p20 handles \u226420 \u00b5L and p300 handles 40/106/250 \u00b5L and the 30 \u00b5L mixes; all end_state checks pass with no overdispense or empty-well aspiration.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "The tip_before_aspirate, no_cross_contamination and tip_dropped_at_end checks all pass. Primers, templates, fragments and assemblies each use new_tip='always' or a fresh tip per well; shared reagents use one tip only from a clean stock.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "Steps 1-33 are carried out in order with the specified volumes and wells, and the manual steps are recorded with protocol.comment. Only the unspecified 20 \u00b5L mix volume in step 24 differs from the reference (1 \u00b5L), which is within the task text.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "pcr_setup", + "score": 1, + "evidence": "Master mix is 106/40/4/2 \u00b5L, then 19 \u00b5L to A1:G1, 2.5 \u00b5L of each primer, and 1 \u00b5L template from the template_src list A1,A1,B1,C1,A1,A1,D1, with fresh tips each time.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dpni_and_cleanup", + "score": 1, + "evidence": "19 \u00b5L water, 5 \u00b5L rCutSmart and 1 \u00b5L DpnI are added to A1:G1 and mixed 3x at 30 \u00b5L. The 37/65\u00b0C incubation and the Zymo clean-up and pause are recorded with comments in order.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "assembly_mix", + "score": 1, + "evidence": "Each assembly well gets 7 \u00b5L water, 2 \u00b5L T4 buffer, 3/2/3 \u00b5L of fragments 2/5/6, 2 \u00b5L of fragments 1/3/4/7 paired to A1-D1, and 1 \u00b5L enzyme mix mixed 5x.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "cycling_and_transformation", + "score": 1, + "evidence": "The Golden Gate program is recorded as a comment (30\u00d737/16\u00b0C, 60\u00b0C, 4\u00b0C), followed by the clean-up comment, 5 \u00b5L transfers to cells A1:D1, the heat-shock comment, 250 \u00b5L LB, and the recovery and plating comments.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "pcr_setup": 1.0, + "dpni_and_cleanup": 1.0, + "assembly_mix": 1.0, + "cycling_and_transformation": 1.0 + }, + "summary": "The protocol closely matches the reference and the task: quantities, wells, order and manual-step comments are all correct. All measured checks pass, and I found no tip, volume or fidelity issues.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "pcr_setup": 0.0625, + "dpni_and_cleanup": 0.0625, + "assembly_mix": 0.0625, + "cycling_and_transformation": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/protocol.py new file mode 100644 index 0000000..89bded8 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/protocol.py @@ -0,0 +1,208 @@ +"""AssemblyTron Golden Gate assembly of four four-fragment chromoprotein plasmids. + +Build four chromoprotein expression plasmids (tsPurple, YukonOFP, aeBlue, +fuGFP) starting from PCR amplification of seven fragments with j5-designed +primers, DpnI digestion, fragment clean-up, Golden Gate assembly with +BsaI-HFv2 + T4 DNA ligase, and transformation into E. coli TOP10. + +Manual steps (thermocycling, column clean-ups, heat shock, recovery, plating) +cannot be simulated and are recorded with protocol.comment() at the point +they occur. +""" + +from opentrons import protocol_api + +metadata = { + 'protocolName': 'AssemblyTron Golden Gate Assembly of Chromoprotein Plasmids', + 'author': 'AssemblyTron', + 'description': 'Four four-fragment chromoprotein expression plasmids by Golden Gate assembly.', + 'apiLevel': '2.13', +} + + +def run(protocol: protocol_api.ProtocolContext): + # ------------------------------------------------------------------ labware + tubes_50ml = protocol.load_labware( + 'opentrons_6_tuberack_falcon_50ml_conical', 1, label='tubes_50ml_1') + tubes_1_5ml = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 3, label='primer_plate') + template_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='template_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 5, label='pcr_plate') + assembly_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='assembly_plate') + cells_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 7, label='cells_plate') + tubes_15ml = protocol.load_labware( + 'opentrons_15_tuberack_falcon_15ml_conical', 8, label='tubes_15ml_1') + + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + # ---------------------------------------------------------------- reagents + water = tubes_50ml['A1'] + q5_buffer = tubes_1_5ml['A1'] + dntp = tubes_1_5ml['B1'] + q5_pol = tubes_1_5ml['C1'] + pcr_mm = tubes_1_5ml['D1'] # PCR master mix prepared in situ + rcutsmart = tubes_1_5ml['A2'] + dpni = tubes_1_5ml['B2'] + t4_buffer = tubes_1_5ml['C2'] + gg_enzyme = tubes_1_5ml['D2'] # BsaI-HFv2 + T4 ligase + lb_dextrose = tubes_15ml['A1'] + + fwd_primer_wells = ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1'] + rev_primer_wells = ['A2', 'B2', 'C2', 'D2', 'E2', 'F2', 'G2'] + pcr_wells = ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1'] + template_src = ['A1', 'A1', 'B1', 'C1', 'A1', 'A1', 'D1'] + assembly_wells = ['A1', 'B1', 'C1', 'D1'] + cells_wells = ['A1', 'B1', 'C1', 'D1'] + + pcr_dest = [pcr_plate[w] for w in pcr_wells] + assembly_dest = [assembly_plate[w] for w in assembly_wells] + cells_dest = [cells_plate[w] for w in cells_wells] + + # 1-4. Prepare the PCR master mix (buffer, dNTPs, polymerase, water) in D1. + p300.transfer(106, water, pcr_mm, new_tip='once') # 106 µL water + p300.transfer(40, q5_buffer, pcr_mm, new_tip='once') # 40 µL 5X Q5 buffer + p20.transfer(4, dntp, pcr_mm, new_tip='once') # 4 µL 10 mM dNTPs + p20.transfer(2, q5_pol, pcr_mm, new_tip='once') # 2 µL Q5 polymerase + + # 5. Mix the master mix. + p300.pick_up_tip() + p300.mix(5, 100, pcr_mm) + p300.drop_tip() + + # 6. Distribute master mix to the seven PCR wells. + p20.transfer(19, pcr_mm, pcr_dest, new_tip='once') + + # 7. Forward primers (1 µM). + p20.transfer(2.5, [primer_plate[w] for w in fwd_primer_wells], + pcr_dest, new_tip='always') + + # 8. Reverse primers (1 µM). + p20.transfer(2.5, [primer_plate[w] for w in rev_primer_wells], + pcr_dest, new_tip='always') + + # 9. Linearized template plasmid (0.5 ng/µL), 1 µL per fragment. + p20.transfer(1, [template_plate[w] for w in template_src], + pcr_dest, new_tip='always') + + # 10. Mix the PCR reactions. + for w in pcr_wells: + p20.pick_up_tip() + p20.mix(3, 15, pcr_plate[w]) + p20.drop_tip() + + # 11. Manual: gradient PCR in the Bio-Rad C100 thermocycler. + protocol.comment( + 'Seal pcr_plate (or move the PCR tubes) and transfer manually to the ' + 'Bio-Rad C100 gradient thermocycler. Run: 98°C 30 s; 34 cycles of ' + '98°C 10 s, annealing 30 s at the AssemblyTron/j5 optimal-gradient ' + 'temperature for each fragment, 72°C extension at the time set by ' + 'AssemblyTron (about 20-30 s/kb); final extension 72°C 5 min; hold ' + '4°C. Take a sample of each reaction for gel electrophoresis, then ' + 'return to the OT-2.') + + # 12-14. DpnI digestion master (water, rCutSmart, DpnI) into each well. + p20.transfer(19, water, pcr_dest, new_tip='once') + p20.transfer(5, rcutsmart, pcr_dest, new_tip='once') + p20.transfer(1, dpni, pcr_dest, new_tip='once') + + # 15. Mix the DpnI reactions. + for w in pcr_wells: + p300.pick_up_tip() + p300.mix(3, 30, pcr_plate[w]) + p300.drop_tip() + + # 16. Manual: DpnI digestion / inactivation. + protocol.comment( + 'Incubate pcr_plate A1:G1 at 37°C for 30 min, then 65°C for 20 min ' + '(DpnI inactivation) on the thermocycler block.') + + # 17. Manual: clean and concentrate the fragments. + protocol.comment( + 'Pause the protocol. Clean and concentrate each of the 7 fragments ' + 'with a Zymo DNA Clean & Concentrator-5 column: 5:1 DNA Binding ' + 'Buffer to sample (250 µL per 50 µL), spin 30 s, wash 2 x 200 µL DNA ' + 'Wash Buffer (30 s spins), elute in 20 µL water after 1 min at room ' + 'temperature (30 s spin). Return the eluted fragments to their ' + 'original positions pcr_plate A1:G1 (fragments 1-7) and resume the ' + 'protocol.') + + # 18-19. Set up the Golden Gate reactions: water and ligase buffer. + p20.transfer(7, water, assembly_dest, new_tip='once') + p20.transfer(2, t4_buffer, assembly_dest, new_tip='once') + + # 20-22. Backbone fragments: fragment 2 (backbone), 5 (backbone/KanR), 6 (backbone). + p20.transfer(3, pcr_plate['B1'], assembly_dest, new_tip='once') + p20.transfer(2, pcr_plate['E1'], assembly_dest, new_tip='once') + p20.transfer(3, pcr_plate['F1'], assembly_dest, new_tip='once') + + # 23. Chromoprotein fragments 1, 3, 4, 7 -> one per assembly. + p20.transfer(2, [pcr_plate[w] for w in ['A1', 'C1', 'D1', 'G1']], + assembly_dest, new_tip='always') + + # 24. Golden Gate enzyme mix; mix 5 times after each dispense. + for w in assembly_wells: + p20.pick_up_tip() + p20.aspirate(1, gg_enzyme) + p20.dispense(1, assembly_plate[w]) + p20.mix(5, 20, assembly_plate[w]) + p20.drop_tip() + + # 25. Manual: Golden Gate thermocycling program. + protocol.comment( + 'Run Golden Gate program on assembly_plate in the Opentrons ' + 'thermocycler module: 30 cycles of 37°C 5 min then 16°C 5 min; then ' + '60°C 5 min; hold at 4°C. Lid about 85°C. If no module is available, ' + 'pause and move reactions to another thermocycler.') + + # 26. Manual: clean and concentrate the assemblies. + protocol.comment( + 'Clean and concentrate each assembly with a Zymo DNA Clean & ' + 'Concentrator-5 column (5:1 binding buffer, 2 x 200 µL wash) and ' + 'elute in 10 µL molecular grade water; return the eluates to ' + 'assembly_plate A1:D1.') + + # 27. Add purified assemblies to competent cells. + p20.transfer(5, [assembly_plate[w] for w in assembly_wells], + cells_dest, new_tip='always') + + # 28. Manual: quantify the remaining eluate. + protocol.comment( + 'Measure DNA concentration of the remaining 5 µL of each eluate with ' + 'a NanoDrop-2000c for CFU/µg calculation.') + + # 29. Manual: heat shock. + protocol.comment( + 'Incubate cells_plate A1:D1 for 30 min on ice, heat shock at 42°C ' + 'for 60 s, then return to ice for about 2 min.') + + # 30. Add LB + 0.2% dextrose for recovery. + p300.transfer(250, lb_dextrose, cells_dest, new_tip='once') + + # 31. Manual: recovery. + protocol.comment( + 'Recover cells at 37°C for 60 min with shaking (about 250 rpm).') + + # 32. Manual: plating. + protocol.comment( + 'Plate 50-200 µL of each recovery (neat or a 10x dilution, depending ' + 'on predicted efficiency) onto separate LB agar plates containing ' + 'kanamycin 50 µg/mL; incubate at 37°C overnight.') + + # 33. Manual: colony scoring. + protocol.comment( + 'Manually count colonies per plate and score the fraction showing ' + 'the expected chromoprotein colour (purple, orange, blue, green); ' + 'report CFU/µg of DNA plated.') \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/reward.json new file mode 100644 index 0000000..333077c --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_pcr_setup": 1.0, "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 1.0, "rubric_cycling_and_transformation": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/trial.json new file mode 100644 index 0000000..5ab907d --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/golden-gate-assembly/trial.json @@ -0,0 +1,31 @@ +{ + "task": "golden-gate-assembly", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + }, + "exception": null, + "cost_usd": 0.204623, + "input_tokens": 281989, + "output_tokens": 21557, + "started_at": "2026-10-07T18:21:21.586426Z", + "finished_at": "2026-10-07T18:26:05.285251Z", + "cache_tokens": 37888, + "claude_code_cost_usd": 1.7783739999999997 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/grader.json new file mode 100644 index 0000000..1816cc3 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/grader.json @@ -0,0 +1,234 @@ +{ + "variant": "reference", + "judge_model": "claude-sonnet-5-5", + "suspicious_tokens": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1279 + }, + "checks": { + "checks": [ + { + "name": "48_samples_to_odd_columns", + "pass": true, + "detail": "48 tubes -> 48 wells, odd columns only: True" + }, + { + "name": "step_order", + "pass": true, + "detail": "expected per well: beads > isopropanol > sample > remove > ethanol > remove > ethanol > remove > elution > recover" + }, + { + "name": "binding_volumes_40_250_250", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "mix_5x_after_sample", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "incubation_5min_before_magnet", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_4min_before_first_removal", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "supernatant_removed_each_step", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_engaged_for_all_removals", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "two_500ul_ethanol_washes", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "air_dry_4min", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_100ul", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_off_magnet_then_90s_on", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "recover_70_100ul_one_well_each", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "distinct_elution_wells", + "pass": true, + "detail": "48 distinct elution wells" + }, + { + "name": "elution_plate_4C_before_recovery", + "pass": true, + "detail": "temperatures set: [(0.0, 4.0)]" + }, + { + "name": "fresh_tip_per_sample_no_cross_contact", + "pass": true, + "detail": "48 distinct tips for 48 sample transfers; violations: []" + }, + { + "name": "reservoir_columns_within_15ml", + "pass": true, + "detail": "uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 12000, 10: 12000, 11: 12000, 12: 12000" + } + ], + "checks_passed": 17, + "checks_total": 17, + "total_delay_min": 15.0, + "example_well": "A1", + "example_sequence": [ + "beads 40uL", + "isopropanol 250uL", + "sample 250uL", + "remove 540uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "elution 100uL", + "recover 100uL (magnet)" + ] + }, + "critical_failures": [], + "judge": { + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "Valid apiLevel 2.15, all volumes within pipette ranges (p1000 250/500 \u00b5L, p300 40/100/\u2264190 \u00b5L), reservoir columns \u226415 mL, waste wells \u22641540 \u00b5L, simulator passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 0, + "evidence": "One p1000 tip is reused for 48 wells for isopropanol and for both 500 \u00b5L ethanol dispenses at default well-bottom height into sample wells, and one p300 tip dispenses elution buffer into all six columns, so tips contact sample-bead wells and then move to others.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "The 100 \u00b5L elution buffer is dispensed with no mixing/resuspension of the beads, and shared tips are used across sample wells, so it is not a fully faithful, sound implementation.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "sample_handling", + "score": 1, + "evidence": "Checks '48_samples_to_odd_columns' and 'distinct_elution_wells' pass; the code maps tubes 1\u201348 to odd-column wells with a fresh p1000 tip each.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding_and_separation", + "score": 1, + "evidence": "Checks for 40/250/250 volumes, 5x mix, 5 min incubation, 4 min magnet and removal with magnet engaged all pass 48/48.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "washes_and_drying", + "score": 1, + "evidence": "Checks two_500ul_ethanol_washes, magnet_engaged_for_all_removals and air_dry_4min pass; code disengages the magnet before elution.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "elution_recovery", + "score": 0, + "evidence": "Step 7 only aspirates/dispenses 100 \u00b5L elution buffer into each column with no mixing afterward, so the beads are not resuspended as the rubric requires, though 90 s magnet and 100 \u00b5L recovery to distinct wells pass.", + "votes": [ + 0, + 0, + 0 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 0.0, + "fidelity_to_paper": 0.0, + "sample_handling": 1.0, + "binding_and_separation": 1.0, + "washes_and_drying": 1.0, + "elution_recovery": 0.0 + }, + "summary": "The protocol meets the measured checks on sample placement, binding, washes, drying and recovery volume/timing, and its OT-2 practice is sound. It loses the tip-contamination item because single tips are shared across sample wells for isopropanol, ethanol and elution buffer. It loses the elution item and fidelity because the elution buffer is never mixed to resuspend the beads.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"Valid apiLevel 2.15, all volumes within pipette ranges (p1000 250/500 \\u00b5L, p300 40/100/\\u2264190 \\u00b5L), reservoir columns \\u226415 mL, waste wells \\u22641540 \\u00b5L, simulator passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 0, \"evidence\": \"One p1000 tip is reused for 48 wells for isopropanol and for both 500 \\u00b5L ethanol dispenses at default well-bottom height into sample wells, and one p300 tip dispenses elution buffer into all six columns, so tips contact sample-bead wells and then move to others.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"The 100 \\u00b5L elution buffer is dispensed with no mixing/resuspension of the beads, and shared tips are used across sample wells, so it is not a fully faithful, sound implementation.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks '48_samples_to_odd_columns' and 'distinct_elution_wells' pass; the code maps tubes 1\\u201348 to odd-column wells with a fresh p1000 tip each.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks for 40/250/250 volumes, 5x mix, 5 min incubation, 4 min magnet and removal with magnet engaged all pass 48/48.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Checks two_500ul_ethanol_washes, magnet_engaged_for_all_removals and air_dry_4min pass; code disengages the magnet before elution.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"Step 7 only aspirates/dispenses 100 \\u00b5L elution buffer into each column with no mixing afterward, so the beads are not resuspended as the rubric requires, though 90 s magnet and 100 \\u00b5L recovery to distinct wells pass.\"}], \"summary\": \"The protocol meets the measured checks on sample placement, binding, washes, drying and recovery volume/timing, and its OT-2 practice is sound. It loses the tip-contamination item because single tips are shared across sample wells for isopropanol, ethanol and elution buffer. It loses the elution item and fidelity because the elution buffer is never mixed to resuspend the beads.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.15, all volumes within pipette ranges (p1000 for 250/500 \\u00b5L, p300 multi for \\u2264190 \\u00b5L chunks), reservoir columns \\u226415 mL per the check, and the simulator passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 0, \"evidence\": \"One p1000 tip is reused for all 48 wells of the ethanol washes and the isopropanol step, dispensing at the well bottom (default) into wells that hold sample or bead residue and then going back to the stock. One multichannel tip is reused for elution buffer across columns, which risks carryover between samples and stocks.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"No resuspension mix after adding the 100 \\u00b5L elution buffer, although the paper says the beads are resuspended in it. The code only dispenses the elution buffer and then waits 30 s before engaging the magnet.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"The check 48_samples_to_odd_columns passed. Each tube i goes to mag_wells[i] in columns 1, 3, 5, 7, 9, 11, and the check reports 48 distinct elution wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads, then isopropanol, then sample, then mix(5, 250), then a 5 min delay, then magnet engaged for 4 min, then 540 \\u00b5L removed to waste. The binding, mix, incubation and magnet checks all passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes are each removed with the magnet engaged. A 4 min delay follows with the magnet still on, then mag_mod.disengage(). The checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"The elution buffer is dispensed off-magnet but is never mixed. Only the dispense, a 30 s wait and engage follow. The rubric requires the elution to be mixed, although the 90 s magnet time and the 100 \\u00b5L recovery to distinct wells are correct.\"}], \"summary\": \"The protocol passes all 17 measured checks and gets the plate layout, binding, separation, washes and drying right. It fails on tip handling, where single tips are reused across many sample wells during the ethanol, isopropanol and elution additions. It also fails on fidelity and elution because the beads are never mixed after the 100 \\u00b5L elution buffer is added.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.15; p1000 handles 250/500 \\u00b5L and p300 multi handles 40/100/\\u2264190 \\u00b5L within range; reservoir columns stay under 15 mL (check passed); modules and labware used as intended.\"}, {\"id\": \"tips_and_contamination\", \"score\": 0, \"evidence\": \"One p1000 tip is reused for all 48 ethanol dispenses into sample wells at default bottom height (also the p300 tip for elution buffer across columns), so a tip that touches one sample's well liquid goes on to other samples and back to the reservoir.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"There is no resuspension or mixing after the 100 \\u00b5L elution buffer is added (the paper says beads are resuspended), and the shared-tip handling is a deviation, so the protocol is not fully faithful.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed; mag_wells covers odd columns 1,3,...,11, and distinct_elution_wells passed (48 distinct elution wells).\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Code adds beads 40, isopropanol 250, then sample 250 with mix(5,250), then delay 5 min, engage, delay 4 min, and removes 540 \\u00b5L to waste with the magnet on; the checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes, each removed with the magnet engaged, then a 4 min delay and mag_mod.disengage() before elution; the checks passed.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"The 100 \\u00b5L elution buffer is dispensed off-magnet but never mixed (no mix call before the 30 s delay and re-engage), though the rubric requires it be added and mixed.\"}], \"summary\": \"The protocol gets the sample layout, binding, separation, washes, drying and timings right, and all 17 run-log checks passed. It fails on tip hygiene because one tip is shared across sample wells for the ethanol washes and the elution buffer. It also never mixes the elution buffer to resuspend the beads, so fidelity and elution recovery fail.\"}]", + "level": "hard", + "score": 0.4375, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "sample_handling": 0.0625, + "binding_and_separation": 0.0625, + "washes_and_drying": 0.0625, + "elution_recovery": 0.0625 + }, + "model": "anthropic/claude-sonnet-5.5", + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "rewards": { + "reward": 0.4375, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 0.4375, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 0.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/protocol.py new file mode 100644 index 0000000..68a4512 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/protocol.py @@ -0,0 +1,257 @@ +""" +OT-2 in-house magnetic-bead SARS-CoV-2 RNA extraction (48 samples). + +Implements the "OT-2in-house" protocol described in: + + Lázaro-Perona F., et al. "Evaluation of two automated low-cost RNA + extraction protocols for SARS-CoV-2 detection." PLOS ONE 16(2): e0246302 + (2021). https://doi.org/10.1371/journal.pone.0246302 + +Protocol steps and volumes (from the paper, Table 1 / "Protocol design"): + + 1. Dispense 40 µL magnetic beads + 250 µL isopropanol + 250 µL sample per + well, mix by pipetting five times, incubate 5 min at room temperature. + 2. Activate the GEN1 magnetic module for 4 min. + 3. Collect the supernatant and discard. + 4. Add 500 µL of 70% ethanol, collect and discard (wash 1). + 5. Add 500 µL of 70% ethanol, collect and discard (wash 2). + 6. Air dry for 4 min (magnet still engaged). + 7. Turn off the magnetic module and add 100 µL elution buffer. + 8. After 30 s, turn on the magnetic module. + 9. After 90 s, collect the supernatant and transfer to a 96-well plate. + +The 48 inactivated samples are placed into the odd columns (1, 3, 5, 7, 9, 11) +of the deep-well plate on the magnetic module, one sample per well. Each eluate +is recovered into the matching well of the elution plate held at 4 °C on the +temperature module. +""" + +from opentrons import protocol_api + +metadata = { + "protocolName": "OT-2 in-house magnetic-bead RNA extraction (48 samples)", + "description": ( + "OT-2in-house SARS-CoV-2 RNA extraction per Lázaro-Perona et al. 2021 " + "(PLOS ONE, doi:10.1371/journal.pone.0246302)." + ), + "author": "OT-2", + "apiLevel": "2.15", +} + +# --- Reagent / sample volumes (µL), exactly as in the paper ----------------- +BEADS_VOL = 40 +ISOPROPANOL_VOL = 250 +SAMPLE_VOL = 250 +WASH_VOL = 500 # 70% ethanol, both washes +ELUTION_VOL = 100 + +# --- Timings (paper) -------------------------------------------------------- +BIND_INCUBATION_MIN = 5 # step 1 +MAGNET_BIND_MIN = 4 # step 2 +AIR_DRY_MIN = 4 # step 6 +ELUTION_WAIT_SEC = 30 # step 8 (elution buffer before re-engaging magnet) +ELUTION_MAGNET_SEC = 90 # step 9 (re-engaging before collecting eluate) + +MIX_REPETITIONS = 5 # "Mix by pipetting five times" +MIX_VOLUME = 250 + +# Binding reaction total volume to discard in step 3. +BINDING_SUPERNATANT_VOL = BEADS_VOL + ISOPROPANOL_VOL + SAMPLE_VOL # 540 µL + + +def run(protocol: protocol_api.ProtocolContext): + + # ------------------------------------------------------------------ labware + tips_1000 = protocol.load_labware("opentrons_96_filtertiprack_1000ul", 11) + tips_200_2 = protocol.load_labware("opentrons_96_filtertiprack_200ul", 2) + tips_200_3 = protocol.load_labware("opentrons_96_filtertiprack_200ul", 3) + tips_200_9 = protocol.load_labware("opentrons_96_filtertiprack_200ul", 9) + + # Slot 1: waste plate for discarded supernatant and washes. + waste = protocol.load_labware("usascientific_96_wellplate_2.4ml_deep", 1) + + # Slot 4: GEN1 magnetic module holding the sample/extraction (deep-well) plate. + mag_mod = protocol.load_module("magnetic module", 4) + mag_plate = mag_mod.load_labware("usascientific_96_wellplate_2.4ml_deep") + + # Slot 5: reagent reservoir. + reservoir = protocol.load_labware("nest_12_reservoir_15ml", 5) + + # Slot 6: GEN1 temperature module holding the elution plate. + temp_mod = protocol.load_module("tempdeck", 6) + elu_plate = temp_mod.load_labware("thermo_96_wellplate_200ul") + + # Slots 10 and 7: inactivated samples in 2 mL tubes (24 per rack). + rack_samples_1_24 = protocol.load_labware( + "opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap", 10 + ) + rack_samples_25_48 = protocol.load_labware( + "opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap", 7 + ) + # Slot 12 is the fixed trash. + + # ------------------------------------------------------------------ pipettes + p1000 = protocol.load_instrument( + "p1000_single_gen2", "left", tip_racks=[tips_1000] + ) + p300 = protocol.load_instrument( + "p300_multi_gen2", "right", tip_racks=[tips_200_2, tips_200_3, tips_200_9] + ) + + # Gentle flow rates to avoid disturbing the magnetic bead pellets. + p1000.flow_rate.aspirate = 150 + p1000.flow_rate.dispense = 150 + p300.flow_rate.aspirate = 60 + p300.flow_rate.dispense = 60 + + # ------------------------------------------------------------------ reservoir map + beads_well = reservoir["A2"] # col 2: magnetic beads + elution_buffer = reservoir["A4"] # col 4: elution buffer + iso_wells = [reservoir["A6"], reservoir["A7"]] # col 6-7: isopropanol + etoh_wash1 = [reservoir["A9"], reservoir["A10"]] # col 9-10: 70% ethanol + etoh_wash2 = [reservoir["A11"], reservoir["A12"]] # col 11-12: 70% ethanol + + # ------------------------------------------------------------------ sample layout + odd_columns = [1, 3, 5, 7, 9, 11] + + # 48 samples in order 1..48 (each rack holds 24 tubes, A1..D6). + source_tubes = list(rack_samples_1_24.wells()) + list(rack_samples_25_48.wells()) + + # Sample i -> the matching odd-column well of the magnetic plate (one per well). + mag_wells = [] + for c in odd_columns: + mag_wells.extend(mag_plate.columns()[c - 1]) + + # Split isopropanol/ethanol across a pair of reservoir columns + # (12 mL needed per reagent <= 15 mL per column when halved). + def iso_source(i): + return iso_wells[0] if i < 24 else iso_wells[1] + + def etoh1_source(i): + return etoh_wash1[0] if i < 24 else etoh_wash1[1] + + def etoh2_source(i): + return etoh_wash2[0] if i < 24 else etoh_wash2[1] + + def remove_supernatant(volume, src_col, dst_col): + """Aspirate `volume` µL from a column of 8 wells and discard to waste. + + Uses the p300 multichannel with one set of tips, in <=190 µL chunks so it + never exceeds the 200 µL tip capacity. Tips are picked up / dropped by the + caller. + """ + remaining = volume + while remaining > 0: + vol = min(190, remaining) + p300.aspirate(vol, src_col[0].bottom(2)) + p300.dispense(vol, dst_col[0].top(-5)) + remaining -= vol + + # ------------------------------------------------------------------ setup + # Keep the elution plate at 4 °C (blocks until the module reaches target). + temp_mod.set_temperature(celsius=4) + mag_mod.disengage() + + # ================= Step 1: reagent mix ===================================== + protocol.comment( + "Step 1: add 40 µL beads, 250 µL isopropanol and 250 µL sample per well; " + "mix 5x; incubate 5 min at room temperature." + ) + # 40 µL magnetic beads — one multichannel tip reused (clean reagent), the + # beads are resuspended in the reservoir immediately before each pickup. + p300.pick_up_tip() + for c in odd_columns: + p300.mix(3, BEADS_VOL, beads_well) + p300.aspirate(BEADS_VOL, beads_well) + p300.dispense(BEADS_VOL, mag_plate.columns()[c - 1][0]) + p300.drop_tip() + + # 250 µL isopropanol — one single-channel tip reused (clean reagent). + p1000.pick_up_tip() + for i in range(48): + p1000.aspirate(ISOPROPANOL_VOL, iso_source(i)) + p1000.dispense(ISOPROPANOL_VOL, mag_wells[i]) + p1000.drop_tip() + + # 250 µL sample + mix five times — fresh tip per sample (avoids carry-over). + for i in range(48): + p1000.pick_up_tip() + p1000.aspirate(SAMPLE_VOL, source_tubes[i]) + p1000.dispense(SAMPLE_VOL, mag_wells[i]) + p1000.mix(MIX_REPETITIONS, MIX_VOLUME, mag_wells[i]) + p1000.drop_tip() + + protocol.delay(minutes=BIND_INCUBATION_MIN) + + # ================= Step 2: engage magnet 4 min ============================== + protocol.comment("Step 2: activate the GEN1 magnetic module for 4 min.") + mag_mod.engage() # uses the deep-well plate's default engage height (14.94 mm) + protocol.delay(minutes=MAGNET_BIND_MIN) + + # ================= Step 3: discard binding supernatant ====================== + protocol.comment("Step 3: collect and discard the binding supernatant.") + for k, c in enumerate(odd_columns): + p300.pick_up_tip() + remove_supernatant( + BINDING_SUPERNATANT_VOL, mag_plate.columns()[c - 1], waste.columns()[k] + ) + p300.drop_tip() + + # ================= Step 4: wash 1 (70% ethanol) ============================= + protocol.comment("Step 4: add 500 µL 70% ethanol, collect and discard.") + p1000.pick_up_tip() + for i in range(48): + p1000.aspirate(WASH_VOL, etoh1_source(i)) + p1000.dispense(WASH_VOL, mag_wells[i]) + p1000.drop_tip() + for k, c in enumerate(odd_columns): + p300.pick_up_tip() + remove_supernatant(WASH_VOL, mag_plate.columns()[c - 1], waste.columns()[k]) + p300.drop_tip() + + # ================= Step 5: wash 2 (70% ethanol) ============================= + protocol.comment("Step 5: add 500 µL 70% ethanol, collect and discard.") + p1000.pick_up_tip() + for i in range(48): + p1000.aspirate(WASH_VOL, etoh2_source(i)) + p1000.dispense(WASH_VOL, mag_wells[i]) + p1000.drop_tip() + for k, c in enumerate(odd_columns): + p300.pick_up_tip() + remove_supernatant(WASH_VOL, mag_plate.columns()[c - 1], waste.columns()[k]) + p300.drop_tip() + + # ================= Step 6: air dry 4 min ==================================== + protocol.comment("Step 6: air dry the beads for 4 min (magnet still engaged).") + protocol.delay(minutes=AIR_DRY_MIN) + + # ================= Step 7: disengage + add elution buffer =================== + protocol.comment("Step 7: turn off the magnet and add 100 µL elution buffer.") + mag_mod.disengage() + p300.pick_up_tip() + for c in odd_columns: + p300.aspirate(ELUTION_VOL, elution_buffer) + p300.dispense(ELUTION_VOL, mag_plate.columns()[c - 1][0]) + p300.drop_tip() + + # ================= Step 8: wait 30 s, re-engage ============================= + protocol.comment("Step 8: after 30 s turn on the magnetic module.") + protocol.delay(seconds=ELUTION_WAIT_SEC) + mag_mod.engage() + + # ================= Step 9: wait 90 s, recover eluate ======================== + protocol.comment( + "Step 9: after 90 s collect the supernatant and transfer to the 4 °C plate." + ) + protocol.delay(seconds=ELUTION_MAGNET_SEC) + for k, c in enumerate(odd_columns): + src_col = mag_plate.columns()[c - 1] + dst_col = elu_plate.columns()[c - 1] + p300.pick_up_tip() + p300.aspirate(ELUTION_VOL, src_col[0].bottom(1)) + p300.dispense(ELUTION_VOL, dst_col[0]) + p300.blow_out(dst_col[0]) + p300.drop_tip() + + mag_mod.disengage() + protocol.comment("Extraction complete: 48 eluates on the 4 °C elution plate.") \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/reward.json new file mode 100644 index 0000000..ee468eb --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.4375, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 0.4375, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 0.0, "rubric_fidelity_to_paper": 0.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 0.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/trial.json new file mode 100644 index 0000000..9533d4d --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction-hard/trial.json @@ -0,0 +1,32 @@ +{ + "task": "opentrons-rna-extraction-hard", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.4375, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 0.4375, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 0.0 + }, + "exception": null, + "cost_usd": 0.186434, + "input_tokens": 1601814, + "output_tokens": 53174, + "started_at": "2026-10-07T18:21:21.721531Z", + "finished_at": "2026-10-07T18:29:47.117493Z", + "cache_tokens": 1529856, + "claude_code_cost_usd": 2.454068 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/grader.json new file mode 100644 index 0000000..e6a36d9 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/grader.json @@ -0,0 +1,241 @@ +{ + "variant": "reference", + "judge_model": "claude-sonnet-5-5", + "suspicious_tokens": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1324 + }, + "checks": { + "checks": [ + { + "name": "48_samples_to_odd_columns", + "pass": true, + "detail": "48 tubes -> 48 wells, odd columns only: True" + }, + { + "name": "step_order", + "pass": true, + "detail": "expected per well: beads > isopropanol > sample > remove > ethanol > remove > ethanol > remove > elution > recover" + }, + { + "name": "binding_volumes_40_250_250", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "mix_5x_after_sample", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "incubation_5min_before_magnet", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_4min_before_first_removal", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "supernatant_removed_each_step", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_engaged_for_all_removals", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "two_500ul_ethanol_washes", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "air_dry_4min", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_100ul", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_off_magnet_then_90s_on", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "recover_70_100ul_one_well_each", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "distinct_elution_wells", + "pass": true, + "detail": "48 distinct elution wells" + }, + { + "name": "elution_plate_4C_before_recovery", + "pass": true, + "detail": "temperatures set: [(0.0, 4.0)]" + }, + { + "name": "fresh_tip_per_sample_no_cross_contact", + "pass": true, + "detail": "48 distinct tips for 48 sample transfers; violations: []" + }, + { + "name": "reservoir_columns_within_15ml", + "pass": false, + "detail": "uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 16000, 10: 16000, 11: 8000, 12: 8000" + }, + { + "name": "recover_about_80ul", + "pass": true, + "detail": "70-90 uL of the 100 uL eluate, as the task's step 13 says; 48/48 sample wells" + } + ], + "checks_passed": 17, + "checks_total": 18, + "total_delay_min": 15.0, + "example_well": "A1", + "example_sequence": [ + "beads 40uL", + "isopropanol 250uL", + "sample 250uL", + "remove 540uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "elution 100uL", + "recover 80uL (magnet)" + ] + }, + "critical_failures": [ + "reservoir_columns_within_15ml" + ], + "judge": { + "items": [ + { + "id": "robot_practice", + "score": 0, + "evidence": "Check reservoir_columns_within_15ml failed: ethanol columns 9 and 10 are each drawn 16000 uL (ethanol[i % 4] sends two columns of wash volume to each), exceeding the 15 mL well capacity.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "fresh_tip_per_sample_no_cross_contact passed (48 distinct tips, no violations); p1000 uses a new tip per sample, p300 uses a fresh tip per column for removals, and every tip is dropped.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All 14 steps are in order with correct volumes, delays (5 min, 4 min, 4 min, 30 s, 90 s), magnet states and 4 C setting, and comments match the code.", + "votes": [ + 1, + 0, + 1 + ] + }, + { + "id": "sample_handling", + "score": 1, + "evidence": "48_samples_to_odd_columns and distinct_elution_wells passed; tubes_10 + tubes_7 map one-to-one to the 48 odd-column wells.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding_and_separation", + "score": 1, + "evidence": "The code adds 40 uL beads, 250 uL isopropanol, then 250 uL sample with a 5x mix, a 5 min delay, the magnet engaged for 4 min, and removal of 540 uL with the magnet on; the related checks passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "washes_and_drying", + "score": 1, + "evidence": "Two 500 uL ethanol washes are each removed with the magnet engaged, followed by a 4 min delay with the magnet still engaged and then a disengage before elution; two_500ul_ethanol_washes and air_dry_4min passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "elution_recovery", + "score": 1, + "evidence": "The code disengages, adds 100 uL, mixes, waits 30 s, engages for 90 s, then transfers 80 uL to a distinct well; recover_about_80ul passed.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 0.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "sample_handling": 1.0, + "binding_and_separation": 1.0, + "washes_and_drying": 1.0, + "elution_recovery": 1.0 + }, + "summary": "The protocol follows every step with correct volumes, timings, magnet states and per-sample tip handling, and all but one run-log check passed. It loses robot_practice because the ethanol reservoir columns 9 and 10 are each drawn 16 mL, over the 15 mL well capacity.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 0, \"evidence\": \"Check reservoir_columns_within_15ml failed: ethanol columns 9 and 10 are each drawn 16000 uL (ethanol[i % 4] sends two columns of wash volume to each), exceeding the 15 mL well capacity.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"fresh_tip_per_sample_no_cross_contact passed (48 distinct tips, no violations); p1000 uses a new tip per sample, p300 uses a fresh tip per column for removals, and every tip is dropped.\"}, {\"id\": \"fidelity_to_task\", \"score\": 1, \"evidence\": \"All 14 steps are in order with correct volumes, delays (5 min, 4 min, 4 min, 30 s, 90 s), magnet states and 4 C setting, and comments match the code.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"48_samples_to_odd_columns and distinct_elution_wells passed; tubes_10 + tubes_7 map one-to-one to the 48 odd-column wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"The code adds 40 uL beads, 250 uL isopropanol, then 250 uL sample with a 5x mix, a 5 min delay, the magnet engaged for 4 min, and removal of 540 uL with the magnet on; the related checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 uL ethanol washes are each removed with the magnet engaged, followed by a 4 min delay with the magnet still engaged and then a disengage before elution; two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"The code disengages, adds 100 uL, mixes, waits 30 s, engages for 90 s, then transfers 80 uL to a distinct well; recover_about_80ul passed.\"}], \"summary\": \"The protocol follows every step with correct volumes, timings, magnet states and per-sample tip handling, and all but one run-log check passed. It loses robot_practice because the ethanol reservoir columns 9 and 10 are each drawn 16 mL, over the 15 mL well capacity.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 0, \"evidence\": \"The failed check reservoir_columns_within_15ml shows columns 9 and 10 each drawn 16000 \\u00b5L (ethanol[i % 4] sends 2 columns of washes to A9 and A10), exceeding the 15 mL well capacity.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passes (48 distinct tips); p1000 uses a fresh tip per sample, p300 uses a fresh tip per column for supernatant removal, elution mixing and recovery, and all tips are dropped.\"}, {\"id\": \"fidelity_to_task\", \"score\": 0, \"evidence\": \"Ethanol is planned so that reservoir columns 9 and 10 are asked for 16 mL each, more than the well can hold, so the specified reagent supply is not actually deliverable.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks 48_samples_to_odd_columns and distinct_elution_wells pass; code zips tubes_10+tubes_7 to the 48 wells in odd columns 1,3,...,11.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Code adds 40 \\u00b5L beads, 250 \\u00b5L isopropanol, then 250 \\u00b5L sample with mix(5, 250), delay 5 min, engage, delay 4 min, then removes 2x270 \\u00b5L to waste; the related run-log checks all pass.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two washes of 2x250 \\u00b5L ethanol in and 2x250 \\u00b5L out with the magnet engaged, then a 4 min delay and mag_module.disengage() before elution; checks two_500ul_ethanol_washes and air_dry_4min pass.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"100 \\u00b5L elution is added after disengage and mixed, with a 30 s wait, engage and 90 s wait; then transfer(80) to a distinct elution well, and recover_about_80ul passes.\"}], \"summary\": \"The protocol follows the extraction steps and gets the sample mapping, tip handling, binding, washes and elution right. It fails robot_practice and fidelity because the ethanol distribution draws 16 mL from reservoir columns 9 and 10, above the 15 mL well capacity (the reservoir_columns_within_15ml check failed).\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 0, \"evidence\": \"Check reservoir_columns_within_15ml failed: ethanol columns 9 and 10 are drawn 16000 \\u00b5L each (ethanol[i % 4] reused for columns 1/5 and 3/7 in both washes), exceeding the 15 mL well capacity.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"fresh_tip_per_sample_no_cross_contact passed (48 distinct tips); p1000 gets a new tip per sample, and reagent tips never aspirate from sample wells; all tips dropped.\"}, {\"id\": \"fidelity_to_task\", \"score\": 1, \"evidence\": \"All 14 steps are in order with the right volumes and timings (5 min, 4 min, 2x500 \\u00b5L ethanol, 4 min dry, 100 \\u00b5L elution, 30 s + 90 s, 80 \\u00b5L recovery, final disengage); the reservoir overdraw is a practice issue, not a deviation from the stated steps.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"48_samples_to_odd_columns passed; tubes_10 + tubes_7 map one-to-one onto odd-column wells, and each eluate goes to its own elution well.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Code adds beads 40, isopropanol 250, then 250 \\u00b5L sample with mix(5, 250), delay 5 min, engage + 4 min, then removes 2x270 \\u00b5L to waste with magnet on; the matching checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two washes of 2x250 \\u00b5L ethanol added and 2x250 \\u00b5L removed with the magnet engaged, then 4 min delay with magnet still on, then disengage before elution.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"Disengage, add 100 \\u00b5L, mix 10x80, delay 30 s, engage, delay 90 s, transfer 80 \\u00b5L to a distinct well of the 4 \\u00b0C elution plate (recover_about_80ul passed).\"}], \"summary\": \"The protocol follows the specified steps, volumes and timings and handles tips correctly. It fails robot_practice because the ethanol distribution draws 16 mL from reservoir columns 9 and 10, above the 15 mL well capacity.\"}]", + "level": "easy", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "sample_handling": 0.0625, + "binding_and_separation": 0.0625, + "washes_and_drying": 0.0625, + "elution_recovery": 0.0625 + }, + "model": "anthropic/claude-sonnet-5.5", + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9444, + "judge_score": 0.75, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 1.0, + "deterministic_reward": 0.4722, + "rubric_robot_practice": 0.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/protocol.py new file mode 100644 index 0000000..a41e4b2 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/protocol.py @@ -0,0 +1,203 @@ +"""Magnetic-bead RNA extraction (SARS-CoV-2), in-house OT-2 protocol, 48 samples. + +Implements the "OT-2 in-house" magnetic-bead extraction described in +"Evaluation of two automated low-cost RNA extraction protocols for SARS-CoV-2 +detection" (La'zaro-Perona et al., PLOS ONE 2021, doi:10.1371/journal.pone.0246302). + +Deck layout (operator has loaded the robot as follows): + 1 waste plate (usascientific_96_wellplate_2.4ml_deep) + 2,3 200 uL filter tips + 4 Magnetic Module GEN1 + sample/extraction deep-well plate + 5 12-column reagent reservoir (beads, isopropanol, ethanol, elution) + 6 Temperature Module GEN1 + elution plate (thermo_96_wellplate_200ul) + 7 samples 25-48 (2 mL tubes) + 9 200 uL filter tips + 10 samples 1-24 (2 mL tubes) + 11 1000 uL filter tips + 12 fixed trash +""" + +from opentrons import protocol_api + +metadata = { + "protocolName": "Mag-bead RNA extraction (OT-2 in-house, 48 samples)", + "apiLevel": "2.15", +} + +NUM_SAMPLES = 48 +# Samples occupy the odd columns of the magnetic-module plate, one per well. +ODD_COLUMNS = [1, 3, 5, 7, 9, 11] + +ENGAGE_HEIGHT = 7 # mm above well bottom for the GEN1 magnetic module + + +def run(protocol: protocol_api.ProtocolContext): + # ---------------------------------------------------------------- deck ---- + waste_plate = protocol.load_labware( + "usascientific_96_wellplate_2.4ml_deep", "1" + ) + tips_200_a = protocol.load_labware("opentrons_96_filtertiprack_200ul", "2") + tips_200_b = protocol.load_labware("opentrons_96_filtertiprack_200ul", "3") + tips_200_c = protocol.load_labware("opentrons_96_filtertiprack_200ul", "9") + + mag_module = protocol.load_module("magnetic module", "4") + mag_plate = mag_module.load_labware("usascientific_96_wellplate_2.4ml_deep") + + reservoir = protocol.load_labware("nest_12_reservoir_15ml", "5") + + temp_module = protocol.load_module("tempdeck", "6") + elution_plate = temp_module.load_labware("thermo_96_wellplate_200ul") + + tubes_7 = protocol.load_labware( + "opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap", "7" + ) + tubes_10 = protocol.load_labware( + "opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap", "10" + ) + + tips_1000 = protocol.load_labware("opentrons_96_filtertiprack_1000ul", "11") + # Slot 12 is the fixed trash; nothing is loaded there. + + # ------------------------------------------------------------ pipettes ---- + p1000 = protocol.load_instrument( + "p1000_single_gen2", "left", tip_racks=[tips_1000] + ) + p300 = protocol.load_instrument( + "p300_multi_gen2", "right", tip_racks=[tips_200_a, tips_200_b, tips_200_c] + ) + + # --------------------------------------------------- reagent reservoir ---- + beads = reservoir["A2"] + elution_buffer = reservoir["A4"] + isopropanol_6 = reservoir["A6"] + isopropanol_7 = reservoir["A7"] + ethanol = [reservoir["A9"], reservoir["A10"], reservoir["A11"], reservoir["A12"]] + + # ------------------------------------------- sample & destination wells ---- + # The multi-channel pipette addresses whole columns; anchor each target + # column with its top well (A of that column). + sample_anchors = [mag_plate.columns_by_name()[str(c)][0] for c in ODD_COLUMNS] + waste_anchors = [waste_plate.columns_by_name()[str(c)][0] for c in ODD_COLUMNS] + elution_anchors = [ + elution_plate.columns_by_name()[str(c)][0] for c in ODD_COLUMNS + ] + # Individual wells, in order, for the single-channel 1000 uL pipette. + sample_columns = [mag_plate.columns_by_name()[str(c)] for c in ODD_COLUMNS] + sample_wells = [w for col in sample_columns for w in col] # 48 wells + # Samples 1-24 live in slot 10; samples 25-48 live in slot 7. + source_wells = list(tubes_10.wells()) + list(tubes_7.wells()) # 48 tubes + + # ------------------------------------- startup: disengage magnet + chill --- + mag_module.disengage() + # Keep the elution plate at 4 °C before any eluate reaches it. + temp_module.set_temperature(4) + + # ------------------------------------------------------------- binding ---- + + # 2. Add 40 uL magnetic beads (reservoir column 2) to every sample well. + protocol.comment("2. 40 uL magnetic beads") + p300.pick_up_tip() + for anchor in sample_anchors: + p300.transfer(40, beads, anchor, new_tip="never") + p300.drop_tip() + + # 3. Add 250 uL isopropanol (reservoir columns 6-7) to every sample well. + protocol.comment("3. 250 uL isopropanol") + p300.pick_up_tip() + for i, anchor in enumerate(sample_anchors): + source = isopropanol_6 if i < 3 else isopropanol_7 + p300.transfer(250, source, anchor, new_tip="never") + p300.drop_tip() + + # 4. Add 250 uL of each sample, fresh tip per sample, mix 5 times. + protocol.comment("4. 250 uL sample (fresh tip per sample, mix 5x)") + for source, dest in zip(source_wells, sample_wells): + p1000.pick_up_tip() + p1000.aspirate(250, source) + p1000.dispense(250, dest) + p1000.mix(5, 250, dest) + p1000.drop_tip() + + # 5. Incubate 5 min at room temperature. + protocol.comment("5. Incubate 5 min at room temperature") + protocol.delay(minutes=5) + + # 6. Engage the magnet and wait 4 min. + protocol.comment("6. Engage magnet, wait 4 min") + mag_module.engage(height_from_base=ENGAGE_HEIGHT) + protocol.delay(minutes=4) + + # 7. Remove ~540 uL supernatant (40 + 250 + 250) to the waste plate. + protocol.comment("7. Remove ~540 uL supernatant to waste") + for anchor, waste_anchor in zip(sample_anchors, waste_anchors): + p300.pick_up_tip() + for _ in range(2): # 2 x 270 uL = 540 uL + p300.transfer(270, anchor, waste_anchor, new_tip="never") + p300.drop_tip() + + # 8. Wash 1: add 500 uL 70% ethanol, then remove to waste (magnet engaged). + protocol.comment("8. Wash 1: 500 uL 70% ethanol in, then out") + p300.pick_up_tip() + for i, anchor in enumerate(sample_anchors): + source = ethanol[i % len(ethanol)] + for _ in range(2): # 2 x 250 uL = 500 uL + p300.transfer(250, source, anchor, new_tip="never") + p300.drop_tip() + + for anchor, waste_anchor in zip(sample_anchors, waste_anchors): + p300.pick_up_tip() + for _ in range(2): # remove 500 uL + p300.transfer(250, anchor, waste_anchor, new_tip="never") + p300.drop_tip() + + # 9. Wash 2: add 500 uL 70% ethanol, then remove to waste (magnet engaged). + protocol.comment("9. Wash 2: 500 uL 70% ethanol in, then out") + p300.pick_up_tip() + for i, anchor in enumerate(sample_anchors): + source = ethanol[i % len(ethanol)] + for _ in range(2): # 2 x 250 uL = 500 uL + p300.transfer(250, source, anchor, new_tip="never") + p300.drop_tip() + + for anchor, waste_anchor in zip(sample_anchors, waste_anchors): + p300.pick_up_tip() + for _ in range(2): # remove 500 uL + p300.transfer(250, anchor, waste_anchor, new_tip="never") + p300.drop_tip() + + # 10. Air-dry the beads 4 min with the magnet engaged. + protocol.comment("10. Air-dry 4 min (magnet engaged)") + protocol.delay(minutes=4) + + # ------------------------------------------------------------ elution ----- + + # 11. Disengage and add 100 uL elution buffer; mix to resuspend. + protocol.comment("11. Disengage, add 100 uL elution buffer, resuspend") + mag_module.disengage() + + p300.pick_up_tip() + for anchor in sample_anchors: + p300.transfer(100, elution_buffer, anchor, new_tip="never") + p300.drop_tip() + + for anchor in sample_anchors: # resuspend: fresh tip per column + p300.pick_up_tip() + p300.mix(10, 80, anchor) + p300.drop_tip() + + # 12. Wait 30 s, engage the magnet, wait 90 s. + protocol.comment("12. Wait 30 s, engage magnet, wait 90 s") + protocol.delay(seconds=30) + mag_module.engage(height_from_base=ENGAGE_HEIGHT) + protocol.delay(minutes=1, seconds=30) + + # 13. Transfer 80 uL eluate to the 4 °C elution plate (own well each). + protocol.comment("13. Transfer 80 uL eluate to elution plate (4 °C)") + for anchor, elution_anchor in zip(sample_anchors, elution_anchors): + p300.pick_up_tip() + p300.transfer(80, anchor, elution_anchor, new_tip="never") + p300.drop_tip() + + # 14. Disengage the magnet at the end of the run. + protocol.comment("14. Disengage magnet (end of run)") + mag_module.disengage() \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/reward.json new file mode 100644 index 0000000..341c92f --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.9444, "judge_score": 0.75, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 1.0, "deterministic_reward": 0.4722, "rubric_robot_practice": 0.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/trial.json new file mode 100644 index 0000000..03b3b2c --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/opentrons-rna-extraction/trial.json @@ -0,0 +1,32 @@ +{ + "task": "opentrons-rna-extraction", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9444, + "judge_score": 0.75, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 1.0, + "deterministic_reward": 0.4722, + "rubric_robot_practice": 0.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + }, + "exception": null, + "cost_usd": 1.170359, + "input_tokens": 1685986, + "output_tokens": 44933, + "started_at": "2026-10-07T18:21:21.709374Z", + "finished_at": "2026-10-07T18:33:19.625887Z", + "cache_tokens": 49152, + "claude_code_cost_usd": 9.332071000000001 +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/grader.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/grader.json new file mode 100644 index 0000000..f39ac72 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/grader.json @@ -0,0 +1,137 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 7 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plate", + "pass": true, + "detail": "expected volumes [100.0, 100.0], got [100.0, 100.0] (positions free)" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13 is valid, and the 100 \u00b5L transfers use the p300, which covers that volume. The measured checks for overdispense and empty-well aspiration passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "new_tip='once' gives one tip for a single reagent source, and the tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "The single step, 100 \u00b5L from reservoir A1 to plate A1 and B1, is done as specified. The unused p20 and its rack are not loaded, which does not change the outcome.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "deck_and_hardware", + "score": 1, + "evidence": "The reservoir in slot 1 and plate in slot 2 are loaded with the specified labels. The p300 on the right uses the 300 \u00b5L tip rack in slot 11.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "volumes_and_wells", + "score": 1, + "evidence": "p300.transfer(100, reservoir['A1'], [plate['A1'], plate['B1']]) was used, and the end_state check got [100.0, 100.0] in the plate.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "deck_and_hardware": 1.0, + "volumes_and_wells": 1.0 + }, + "summary": "The protocol simulates cleanly, and every measured check passed. It transfers 100 \u00b5L to each of plate A1 and B1 with the p300 and one tip. It leaves out the unused p20 and its rack, which does not change the outcome.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "deck_and_hardware": 0.125, + "volumes_and_wells": 0.125 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/protocol.py b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/protocol.py new file mode 100644 index 0000000..2a7dd9d --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/protocol.py @@ -0,0 +1,28 @@ +from opentrons import protocol_api + +metadata = { + 'apiLevel': '2.13', + 'protocolName': 'Split 200 uL into two 100 uL wells', +} + +def run(protocol: protocol_api.ProtocolContext): + # Load labware with labels exactly as specified + reservoir = protocol.load_labware('nest_1_reservoir_195ml', 1, label='reservoir') + plate = protocol.load_labware('corning_96_wellplate_360ul_flat', 2, label='plate') + + # Pipettes: p300 on the right (for 100 uL), tips are unlimited + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', + tip_racks=[protocol.load_labware('opentrons_96_tiprack_300ul', 11)], + ) + + # Reset tip racks to indicate tips are unlimited + p300.reset_tipracks() + + # Transfer 100 uL from reservoir A1 to plate A1 and B1 + p300.transfer( + 100, + reservoir['A1'], + [plate['A1'], plate['B1']], + new_tip='once', + ) \ No newline at end of file diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/reward.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/reward.json new file mode 100644 index 0000000..5b16841 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0} diff --git a/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/trial.json b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/trial.json new file mode 100644 index 0000000..fd05323 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/deepseek-v4-pro-0813/split-200ul-two-wells/trial.json @@ -0,0 +1,29 @@ +{ + "task": "split-200ul-two-wells", + "model": "deepseek/deepseek-v4-pro-0813", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + }, + "exception": null, + "cost_usd": 0.022981, + "input_tokens": 49087, + "output_tokens": 532, + "started_at": "2026-10-07T18:21:21.726625Z", + "finished_at": "2026-10-07T18:23:44.340553Z", + "cache_tokens": 16410, + "claude_code_cost_usd": 0.18489 +} diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/a1-a12-100ul/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/a1-a12-100ul/trial.json index 597c9e2..3a4622e 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/a1-a12-100ul/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/a1-a12-100ul/trial.json @@ -19,9 +19,11 @@ "rubric_volumes_and_wells": 1.0 }, "exception": null, - "cost_usd": 0.14462450000000002, + "cost_usd": 0.045447, "input_tokens": 56041, "output_tokens": 752, "started_at": "2026-10-07T13:42:06.659918Z", - "finished_at": "2026-10-07T13:43:18.902754Z" + "finished_at": "2026-10-07T13:43:18.902754Z", + "cache_tokens": 39029, + "claude_code_cost_usd": 0.14462450000000002 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ampure-bead-cleanup/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ampure-bead-cleanup/trial.json index b8628d4..56a7290 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ampure-bead-cleanup/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ampure-bead-cleanup/trial.json @@ -21,9 +21,11 @@ "rubric_recovery": 1.0 }, "exception": null, - "cost_usd": 0.20954699999999998, + "cost_usd": 0.070502, "input_tokens": 60085, "output_tokens": 2975, "started_at": "2026-10-07T13:58:56.205406Z", - "finished_at": "2026-10-07T14:01:31.730219Z" + "finished_at": "2026-10-07T14:01:31.730219Z", + "cache_tokens": 41799, + "claude_code_cost_usd": 0.20954699999999998 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/grader.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/grader.json index fcba1cf..624f0e0 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/grader.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/grader.json @@ -5,6 +5,9 @@ "ok": true, "n_commands": 1458 }, + "not_applicable": [ + "end_state:pcr_plate (the source paper does not determine the exact volumes; composition checked instead)" + ], "checks": [ { "name": "simulator_ran", @@ -42,9 +45,9 @@ "detail": "" }, { - "name": "end_state:pcr_plate", - "pass": false, - "detail": "expected {'A1': 20.0, 'A2': 20.0, 'A3': 20.0, 'A4': 20.0, 'A5': 20.0, 'A6': 20.0, 'A7': 20.0, 'A8': 20.0, 'A9': 20.0, 'A10': 20.0, 'A11': 20.0, 'A12': 20.0, 'B1': 20.0, 'B2': 20.0, 'B3': 20.0, 'B4': 20.0, 'B5': 20.0, 'B6': 20.0, 'B7': 20.0, 'B8': 20.0, 'B9': 20.0, 'B10': 20.0, 'B11': 20.0, 'B12': 20.0, 'C1': 20.0, 'C2': 20.0, 'C3': 20.0, 'C4': 20.0, 'C5': 20.0, 'C6': 20.0, 'C7': 20.0, 'C8': 20.0, 'C9': 20.0, 'C10': 20.0, 'C11': 20.0, 'C12': 20.0, 'D1': 20.0, 'D2': 20.0, 'D3': 20.0, 'D4': 20.0, 'D5': 20.0, 'D6': 20.0, 'D7': 20.0, 'D8': 20.0, 'D9': 20.0, 'D10': 20.0, 'D11': 20.0, 'D12': 20.0, 'E1': 20.0, 'E2': 20.0, 'E3': 20.0, 'E4': 20.0, 'E5': 20.0, 'E6': 20.0, 'E7': 20.0, 'E8': 20.0, 'E9': 20.0, 'E10': 20.0, 'E11': 20.0, 'E12': 20.0, 'F1': 20.0, 'F2': 20.0, 'F3': 20.0, 'F4': 20.0, 'F5': 20.0, 'F6': 20.0, 'F7': 20.0, 'F8': 20.0, 'F9': 20.0, 'F10': 20.0, 'F11': 20.0, 'F12': 20.0, 'G1': 20.0, 'G2': 20.0, 'G3': 20.0, 'G4': 20.0, 'G5': 20.0, 'G6': 20.0, 'G7': 20.0, 'G8': 20.0, 'G9': 20.0, 'G10': 20.0, 'G11': 20.0, 'G12': 20.0, 'H1': 20.0, 'H2': 20.0, 'H3': 20.0, 'H4': 20.0, 'H5': 20.0, 'H6': 20.0, 'H7': 20.0, 'H8': 20.0, 'H9': 20.0, 'H10': 20.0, 'H11': 20.0, 'H12': 20.0}, got {'A1': 10.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0, 'E1': 10.0, 'F1': 10.0, 'G1': 10.0, 'H1': 10.0, 'A2': 10.0, 'B2': 10.0, 'C2': 10.0, 'D2': 10.0, 'E2': 10.0, 'F2': 10.0, 'G2': 10.0, 'H2': 10.0, 'A3': 10.0, 'B3': 10.0, 'C3': 10.0, 'D3': 10.0, 'E3': 10.0, 'F3': 10.0, 'G3': 10.0, 'H3': 10.0, 'A4': 10.0, 'B4': 10.0, 'C4': 10.0, 'D4': 10.0, 'E4': 10.0, 'F4': 10.0, 'G4': 10.0, 'H4': 10.0, 'A5': 10.0, 'B5': 10.0, 'C5': 10.0, 'D5': 10.0, 'E5': 10.0, 'F5': 10.0, 'G5': 10.0, 'H5': 10.0, 'A6': 10.0, 'B6': 10.0, 'C6': 10.0, 'D6': 10.0, 'E6': 10.0, 'F6': 10.0, 'G6': 10.0, 'H6': 10.0, 'A7': 10.0, 'B7': 10.0, 'C7': 10.0, 'D7': 10.0, 'E7': 10.0, 'F7': 10.0, 'G7': 10.0, 'H7': 10.0, 'A8': 10.0, 'B8': 10.0, 'C8': 10.0, 'D8': 10.0, 'E8': 10.0, 'F8': 10.0, 'G8': 10.0, 'H8': 10.0, 'A9': 10.0, 'B9': 10.0, 'C9': 10.0, 'D9': 10.0, 'E9': 10.0, 'F9': 10.0, 'G9': 10.0, 'H9': 10.0, 'A10': 10.0, 'B10': 10.0, 'C10': 10.0, 'D10': 10.0, 'E10': 10.0, 'F10': 10.0, 'G10': 10.0, 'H10': 10.0, 'A11': 10.0, 'B11': 10.0, 'C11': 10.0, 'D11': 10.0, 'E11': 10.0, 'F11': 10.0, 'G11': 10.0, 'H11': 10.0, 'A12': 10.0, 'B12': 10.0, 'C12': 10.0, 'D12': 10.0, 'E12': 10.0, 'F12': 10.0, 'G12': 10.0, 'H12': 10.0}" + "name": "composition:pcr_plate", + "pass": true, + "detail": "" }, { "name": "end_state:primer_plate", @@ -58,7 +61,7 @@ { "id": "robot_practice", "score": 0, - "evidence": "The protocol adds a needless manual well-by-well transfer of all 96 reactions to a different plate plus several pauses, extra handling that puts samples at risk. Otherwise, apiLevel 2.15 is valid and all volumes (1-5 uL) are handled by the p20.", + "evidence": "Valid apiLevel 2.15 and P20 volumes of 1\u20135 \u00b5L are fine, but the protocol tells the user to manually transfer all 96 reactions from the Corning pcr_plate into a different PCR plate. That is needless extra handling that puts samples at risk.", "votes": [ 0, 0, @@ -68,7 +71,7 @@ { "id": "tips_and_contamination", "score": 1, - "evidence": "add_reagent picks up a fresh tip for every transfer and drops it. The tip checks (tip_before_aspirate, tip_dropped_at_end, no_cross_contamination) all pass.", + "evidence": "add_reagent picks up a fresh tip and drops it for every transfer. The tip_before_aspirate, no_cross_contamination and tip_dropped_at_end checks all pass.", "votes": [ 1, 1, @@ -78,7 +81,7 @@ { "id": "fidelity_to_paper", "score": 0, - "evidence": "The protocol invents a manual transfer to a real PCR plate and extra pauses, and assumes a 1.25 uM primer concentration. The paper has no such steps, so it is not fully faithful.", + "evidence": "The protocol invents a manual transfer into a separate 0.2 mL PCR plate, a gel electrophoresis step, and assumed primer dilutions and cycling values. The paper's OT-2 workflow runs the reaction in the destination plate.", "votes": [ 0, 0, @@ -88,7 +91,7 @@ { "id": "reaction_setup", "score": 1, - "evidence": "Each 10 uL reaction gets 5 uL Q5 2x (1x final), 4 uL of its primer pair, then 1 uL colony template, with master mix added first. This is an allowed 10 uL reaction.", + "evidence": "Each 10 \u00b5L well gets 5 \u00b5L 2x Q5 (1x final) first, then 4 \u00b5L primer pair, then 1 \u00b5L colony template last.", "votes": [ 1, 1, @@ -98,7 +101,7 @@ { "id": "sample_mapping", "score": 1, - "evidence": "The loops use primer_plate[destination.well_name] and colony_plate[destination.well_name] over pcr_plate.wells(), giving a one-to-one A1-H12 mapping with no well skipped or reused.", + "evidence": "Primer and colony wells map to the same-named pcr_plate wells (primer_plate[destination.well_name], colony_plate[destination.well_name]) across all 96 wells, each used once.", "votes": [ 1, 1, @@ -108,7 +111,7 @@ { "id": "thermocycling", "score": 1, - "evidence": "The comments record sealing and a Q5 program: 98C 30 s, 30 cycles of 98C 10 s / 65C 20 s / 72C 30 s, a 72C 120 s final extension and a 4C hold.", + "evidence": "The comments record sealing and a Q5 program: 98 \u00b0C for 30 s, then 30 cycles of 98 \u00b0C 10 s, 65 \u00b0C 20 s and 72 \u00b0C 30 s, then 72 \u00b0C for 2 min and a 4 \u00b0C hold.", "votes": [ 1, 1, @@ -124,7 +127,7 @@ "sample_mapping": 1.0, "thermocycling": 1.0 }, - "summary": "The pipetting itself is sound: fresh tips throughout, correct master mix, primer and template order, A1-to-A1 mapping, and a reasonable Q5 cycling program. However, it invents a manual transfer of all reactions to a real PCR plate and several pauses, which fails robot_practice and fidelity_to_paper.", + "summary": "The agent builds correct 10 \u00b5L reactions with the right order, fresh tips and a one-to-one well mapping, and it records a sensible Q5 program. It loses robot_practice and fidelity because it adds an invented manual transfer to a different PCR plate, plus a gel step and assumed primer concentrations.", "level": "hard", "score": 0.5, "weights": { @@ -143,10 +146,10 @@ "rewards": { "reward": 0.5, "sim_pass": 1.0, - "checks_frac": 0.8889, + "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, - "deterministic_reward": 0.3333, + "deterministic_reward": 1.0, "judge_score": 0.5, "critical_fail": 0.0, "rubric_robot_practice": 0.0, diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/reward.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/reward.json index 2df1807..e64415a 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/reward.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/reward.json @@ -1 +1 @@ -{"reward": 0.5, "sim_pass": 1.0, "checks_frac": 0.8889, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.3333, "judge_score": 0.5, "critical_fail": 0.0, "rubric_robot_practice": 0.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} +{"reward": 0.5, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 0.5, "critical_fail": 0.0, "rubric_robot_practice": 0.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/trial.json index ba5d3d5..e3bda6a 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening-hard/trial.json @@ -6,10 +6,10 @@ "rewards": { "reward": 0.5, "sim_pass": 1.0, - "checks_frac": 0.8889, + "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, - "deterministic_reward": 0.3333, + "deterministic_reward": 1.0, "judge_score": 0.5, "critical_fail": 0.0, "rubric_robot_practice": 0.0, @@ -20,9 +20,27 @@ "rubric_thermocycling": 1.0 }, "exception": null, - "cost_usd": 0.5308607500000001, + "cost_usd": 0.175628, "input_tokens": 244828, "output_tokens": 9296, "started_at": "2026-10-07T13:39:39.080139Z", - "finished_at": "2026-10-07T13:44:30.315966Z" + "finished_at": "2026-10-07T13:44:30.315966Z", + "original_rewards": { + "reward": 0.5, + "sim_pass": 1.0, + "checks_frac": 0.8889, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.3333, + "judge_score": 0.5, + "critical_fail": 0.0, + "rubric_robot_practice": 0.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 1.0 + }, + "cache_tokens": 214204, + "claude_code_cost_usd": 0.5308607500000001 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening/trial.json index 3bf14f8..4c62378 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/colony-pcr-screening/trial.json @@ -20,9 +20,11 @@ "rubric_thermocycling": 1.0 }, "exception": null, - "cost_usd": 0.195742, + "cost_usd": 0.06404, "input_tokens": 73235, "output_tokens": 2375, "started_at": "2026-10-07T14:01:31.749158Z", - "finished_at": "2026-10-07T14:03:25.377114Z" + "finished_at": "2026-10-07T14:03:25.377114Z", + "cache_tokens": 55884, + "claude_code_cost_usd": 0.195742 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation-hard/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation-hard/trial.json index 2ee92b6..3b39d4c 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation-hard/trial.json @@ -20,9 +20,11 @@ "rubric_soc_recovery": 1.0 }, "exception": null, - "cost_usd": 0.4586475, + "cost_usd": 0.150522, "input_tokens": 197181, "output_tokens": 6795, "started_at": "2026-10-07T13:58:29.839535Z", - "finished_at": "2026-10-07T14:02:36.943922Z" + "finished_at": "2026-10-07T14:02:36.943922Z", + "cache_tokens": 164100, + "claude_code_cost_usd": 0.4586475 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation/trial.json index 6ede812..9c83130 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/ecoli-heat-shock-transformation/trial.json @@ -20,9 +20,11 @@ "rubric_soc_recovery": 1.0 }, "exception": null, - "cost_usd": 0.14921825, + "cost_usd": 0.047349, "input_tokens": 56632, "output_tokens": 995, "started_at": "2026-10-07T13:44:30.334245Z", - "finished_at": "2026-10-07T13:46:00.368630Z" + "finished_at": "2026-10-07T13:46:00.368630Z", + "cache_tokens": 39929, + "claude_code_cost_usd": 0.14921825 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly-hard/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly-hard/trial.json index 90cfb56..5fcbbb1 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly-hard/trial.json @@ -21,9 +21,11 @@ "rubric_cycling_and_transformation": 1.0 }, "exception": null, - "cost_usd": 0.7440309999999999, + "cost_usd": 0.245584, "input_tokens": 358635, "output_tokens": 13285, "started_at": "2026-10-07T13:34:52.817517Z", - "finished_at": "2026-10-07T13:42:06.642645Z" + "finished_at": "2026-10-07T13:42:06.642645Z", + "cache_tokens": 318177, + "claude_code_cost_usd": 0.7440309999999999 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly/trial.json index 10361c2..abab06c 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/golden-gate-assembly/trial.json @@ -21,9 +21,11 @@ "rubric_cycling_and_transformation": 1.0 }, "exception": null, - "cost_usd": 0.38147100000000006, + "cost_usd": 0.132945, "input_tokens": 97215, "output_tokens": 7606, "started_at": "2026-10-07T13:34:52.845480Z", - "finished_at": "2026-10-07T13:39:39.055629Z" + "finished_at": "2026-10-07T13:39:39.055629Z", + "cache_tokens": 72392, + "claude_code_cost_usd": 0.38147100000000006 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/grader.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/grader.json index 98c4646..d131bef 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/grader.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/grader.json @@ -118,7 +118,7 @@ { "id": "robot_practice", "score": 1, - "evidence": "apiLevel 2.13, volumes split into \u2264200 \u00b5L strokes on the p300 multi (min 20 \u00b5L), the p1000 handles 250 \u00b5L samples, no well or reservoir overfilled (check reservoir_columns_within_15ml passes), and the simulator ran clean.", + "evidence": "apiLevel 2.13; p1000 handles 250/540 uL mixes, multi uses \u2264200 uL strokes with 200 uL filter tips, reservoir draws are within 15 mL (check passed), and the 540 uL/1540 uL wells fit the 2.4 mL deep wells.", "votes": [ 1, 1, @@ -128,7 +128,7 @@ { "id": "tips_and_contamination", "score": 1, - "evidence": "The check fresh_tip_per_sample_no_cross_contact passes with 48 distinct tips. Every pick_up_tip is followed by a drop_tip. Removal and elution use fresh tips per column, and the shared reagent tips dispense at well tops without touching liquid.", + "evidence": "Check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips; every removal and elution step uses a fresh tip per column; shared reagent tips only dispense at the well top and never touch sample; all tips are dropped.", "votes": [ 1, 1, @@ -137,18 +137,18 @@ }, { "id": "fidelity_to_paper", - "score": 0, - "evidence": "The eluate recovery aspirates and moves the full ELUTION_VOLUME (100 \u00b5L) from a well holding 100 \u00b5L, with the magnet engaged. The expected recovery is about 80 \u00b5L, so this deviates from the specification.", + "score": 1, + "evidence": "All 17 checks passed; the order of beads, isopropanol, sample, 5 mixes, 5 min wait, 4 min magnet, two 500 uL ethanol washes, 4 min dry, 100 uL elution, 30 s off-magnet, 90 s magnet and recovery matches the paper, and comments are accurate.", "votes": [ - 0, 1, - 0 + 1, + 1 ] }, { "id": "sample_handling", "score": 1, - "evidence": "The check 48_samples_to_odd_columns passes. The p1000 moves 250 \u00b5L from each tube into its own odd-column well, and the 48 eluates land in distinct wells.", + "evidence": "Check 48_samples_to_odd_columns passed, and the single-channel loop moves 250 uL from each tube to its own well in the odd columns, with 48 distinct elution wells.", "votes": [ 1, 1, @@ -158,7 +158,7 @@ { "id": "binding_and_separation", "score": 1, - "evidence": "Beads 40, isopropanol 250, then sample 250 \u00b5L are added in that order, with single.mix(5,...) per sample. A 5 min delay is followed by magnet engage and a 4 min delay, then supernatant is removed with the magnet on (all related checks pass).", + "evidence": "Checks binding_volumes_40_250_250, mix_5x_after_sample, incubation_5min_before_magnet, magnet_4min_before_first_removal and magnet_engaged_for_all_removals passed; the 540 uL supernatant goes to waste with the magnet on.", "votes": [ 1, 1, @@ -168,7 +168,7 @@ { "id": "washes_and_drying", "score": 1, - "evidence": "Two 500 \u00b5L ethanol washes are each removed to waste with the magnet engaged. A 4 min delay follows, then mag.disengage() before elution (checks two_500ul_ethanol_washes and air_dry_4min pass).", + "evidence": "Check two_500ul_ethanol_washes passed, each wash is removed with the magnet engaged, and a 4 min delay precedes mag.disengage() (air_dry_4min passed).", "votes": [ 1, 1, @@ -177,28 +177,28 @@ }, { "id": "elution_recovery", - "score": 0, - "evidence": "Recovery is the full 100 \u00b5L (aspirate ELUTION_VOLUME from source.bottom(z=0.5)), not about 80 \u00b5L. This risks drawing beads and does not match the expected recovery volume.", + "score": 1, + "evidence": "100 uL elution is added after mag.disengage() and mixed, followed by 30 s, magnet on, 90 s, then 100 uL recovered to distinct wells of the 4 C plate (checks elution_off_magnet_then_90s_on, recover_70_100ul_one_well_each and elution_plate_4C_before_recovery passed).", "votes": [ - 0, 1, - 0 + 1, + 1 ] } ], "scores": { "robot_practice": 1.0, "tips_and_contamination": 1.0, - "fidelity_to_paper": 0.0, + "fidelity_to_paper": 1.0, "sample_handling": 1.0, "binding_and_separation": 1.0, "washes_and_drying": 1.0, - "elution_recovery": 0.0 + "elution_recovery": 1.0 }, - "summary": "The protocol follows the paper's step order, timings, magnet use, tip handling and sample-to-well mapping, and all 17 measured checks pass. It recovers the full 100 \u00b5L of eluate instead of about 80 \u00b5L, so elution_recovery and the all-or-nothing fidelity item score 0.", - "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13, volumes split into \\u2264200 \\u00b5L strokes on the p300 multi (min 20 \\u00b5L), the p1000 handles 250 \\u00b5L samples, no well or reservoir overfilled (check reservoir_columns_within_15ml passes), and the simulator ran clean.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The check fresh_tip_per_sample_no_cross_contact passes with 48 distinct tips. Every pick_up_tip is followed by a drop_tip. Removal and elution use fresh tips per column, and the shared reagent tips dispense at well tops without touching liquid.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"The eluate recovery aspirates and moves the full ELUTION_VOLUME (100 \\u00b5L) from a well holding 100 \\u00b5L, with the magnet engaged. The expected recovery is about 80 \\u00b5L, so this deviates from the specification.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"The check 48_samples_to_odd_columns passes. The p1000 moves 250 \\u00b5L from each tube into its own odd-column well, and the 48 eluates land in distinct wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40, isopropanol 250, then sample 250 \\u00b5L are added in that order, with single.mix(5,...) per sample. A 5 min delay is followed by magnet engage and a 4 min delay, then supernatant is removed with the magnet on (all related checks pass).\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes are each removed to waste with the magnet engaged. A 4 min delay follows, then mag.disengage() before elution (checks two_500ul_ethanol_washes and air_dry_4min pass).\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"Recovery is the full 100 \\u00b5L (aspirate ELUTION_VOLUME from source.bottom(z=0.5)), not about 80 \\u00b5L. This risks drawing beads and does not match the expected recovery volume.\"}], \"summary\": \"The protocol follows the paper's step order, timings, magnet use, tip handling and sample-to-well mapping, and all 17 measured checks pass. It recovers the full 100 \\u00b5L of eluate instead of about 80 \\u00b5L, so elution_recovery and the all-or-nothing fidelity item score 0.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; the p1000 handles 250 uL and the p300 multi handles at most 200 uL per stroke. The 17 measured checks pass. Well and reservoir volumes are within capacity (waste 1540 uL/well, reservoir draws under 15 mL).\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips. Each removal uses fresh tips per column and every tip is dropped. Shared reagent tips only dispense at well tops without contacting liquid, and the beads are dispensed into empty wells.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"Step order, volumes (40/250/250, two 500 uL washes, 100 uL elution), 5 min incubation, 4 min magnet, 4 min dry, 30 s off-magnet and 90 s on-magnet delays all match the paper. The elution plate is held at 4 C and the comments are accurate.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed. The single-channel pipette moves 250 uL from each tube to its own well in columns 1,3,...,11 with a fresh tip per sample.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40 uL, then isopropanol 250 uL, then sample 250 uL with mix(5). Then delay(5 min), engage, delay(4 min), and remove_to_waste(540) with the magnet on. Checks mix_5x, incubation_5min, magnet_4min and supernatant_removed pass.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 uL ethanol washes, each removed to waste with the magnet engaged. A 4 min delay follows, then mag.disengage() before elution. Checks two_500ul_ethanol_washes and air_dry_4min pass.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"100 uL elution is added off-magnet and mixed 5x. The code then waits 30 s, engages the magnet for 90 s, and recovers to a distinct well of the 4 C elution plate. It recovers the full 100 uL rather than about 80, but the check recover_70_100ul passed.\"}], \"summary\": \"The protocol passed all 17 measured run-log checks. It follows the paper's step order, volumes, timings and magnet handling with fresh tips for each sample and each removal step. The one minor deviation is recovering the full 100 uL eluate instead of about 80 uL, which the measured check accepts.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13, all volumes within p300 multi (\\u226520, \\u2264200 per stroke) and p1000 ranges; waste 1540 \\u00b5L/well fits 2.4 mL; reservoir draws within 15 mL; 17/17 checks passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed (48 distinct tips); every pick_up_tip is paired with drop_tip; reagent tips dispense at well top without contacting liquid.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"Recovers 100 \\u00b5L from the bead well (`ELUTION_VOLUME` in the recovery aspirate) instead of about 80 \\u00b5L as in the paper-based method, so it is not fully faithful.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed; single-channel one tip per sample from tubes to sample_wells in odd columns, with distinct elution wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40, isopropanol 250, then sample 250, mix(5,400), delay 5 min, magnet engaged with 4 min delay, then removal to waste with magnet on; checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes each removed with the magnet engaged, then 4 min delay and mag.disengage() before elution; checks passed.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"Recovery aspirates and dispenses ELUTION_VOLUME (100 \\u00b5L) rather than about 80 \\u00b5L, even though elution, 90 s magnet and distinct wells at 4 C are right.\"}], \"summary\": \"The protocol follows the paper's order, volumes, timings, magnet use and tip handling, and all 17 measured checks passed. It recovers 100 \\u00b5L of eluate rather than about 80 \\u00b5L, so it fails elution_recovery and, as a deviation from the paper, fidelity_to_paper.\"}]", + "summary": "The protocol passed all 17 measured run-log checks and follows the paper's step order, volumes and timings. Tip handling and labware use are sound, so every rubric item scores 1.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; p1000 handles 250/540 uL mixes, multi uses \\u2264200 uL strokes with 200 uL filter tips, reservoir draws are within 15 mL (check passed), and the 540 uL/1540 uL wells fit the 2.4 mL deep wells.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips; every removal and elution step uses a fresh tip per column; shared reagent tips only dispense at the well top and never touch sample; all tips are dropped.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"All 17 checks passed; the order of beads, isopropanol, sample, 5 mixes, 5 min wait, 4 min magnet, two 500 uL ethanol washes, 4 min dry, 100 uL elution, 30 s off-magnet, 90 s magnet and recovery matches the paper, and comments are accurate.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed, and the single-channel loop moves 250 uL from each tube to its own well in the odd columns, with 48 distinct elution wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks binding_volumes_40_250_250, mix_5x_after_sample, incubation_5min_before_magnet, magnet_4min_before_first_removal and magnet_engaged_for_all_removals passed; the 540 uL supernatant goes to waste with the magnet on.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Check two_500ul_ethanol_washes passed, each wash is removed with the magnet engaged, and a 4 min delay precedes mag.disengage() (air_dry_4min passed).\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"100 uL elution is added after mag.disengage() and mixed, followed by 30 s, magnet on, 90 s, then 100 uL recovered to distinct wells of the 4 C plate (checks elution_off_magnet_then_90s_on, recover_70_100ul_one_well_each and elution_plate_4C_before_recovery passed).\"}], \"summary\": \"The protocol passed all 17 measured run-log checks and follows the paper's step order, volumes and timings. Tip handling and labware use are sound, so every rubric item scores 1.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; multi strokes are capped at 200 uL (\\u226540 uL), the p1000 handles 250 uL and mixes at 400 uL, no well overfills (waste 1540 uL, reservoir draws \\u226412 mL), and the reservoir_columns_within_15ml check passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The check fresh_tip_per_sample_no_cross_contact found 48 distinct tips for the 48 samples. Every removal and elution step uses a fresh tip per column, and every tip is dropped. Reagent tips are shared across columns only by dispensing near the well top, never into liquid.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"All 17 run-log checks pass: 40/250/250 uL in order, 5 mixes, 5 min incubation, 4 min magnet, two 500 uL ethanol washes, 4 min dry, 100 uL elution, 30 s off-magnet, 90 s on-magnet, recovery, 4 C. No missing or reordered steps.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks 48_samples_to_odd_columns and distinct_elution_wells passed. A single-channel pipette with a fresh tip per sample moves each tube to its own odd-column well.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40 uL, isopropanol 250 uL, then sample 250 uL with mix(5, 400). Then delay 5 min, mag.engage, delay 4 min, and removal of 540 uL to waste with the magnet on; the matching checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 uL ethanol washes, each removed with the magnet engaged, then a 4 min delay and mag.disengage() before elution; checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"Adds 100 uL elution off-magnet and mixes 5 times, waits 30 s, engages the magnet for 90 s, then recovers 100 uL into a distinct well of the elution plate held at 4 C; the matching checks passed.\"}], \"summary\": \"The protocol passed all 17 measured run-log checks and matches the paper's steps, volumes, timings and magnet usage. Tip handling is sound, with a fresh tip per sample and per removal. Reagent tips are shared across columns only for non-contact dispenses.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13, all volumes are within pipette ranges (p300 multi strokes \\u2264200 \\u00b5L, p1000 single 250 \\u00b5L), wells stay within capacity (waste 1540 \\u00b5L, reservoir columns \\u226415 mL), and all 17 run-log checks passed.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The check 'fresh_tip_per_sample_no_cross_contact' passed with 48 distinct tips. Every pick_up_tip is followed by drop_tip. Removal, elution and recovery use a fresh multi tip per column, and shared reagent tips dispense at the well top without touching liquid.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 1, \"evidence\": \"The code follows the paper's order and values: beads 40, isopropanol 250, sample 250, mix 5x, 5 min, magnet 4 min, discard, two 500 \\u00b5L ethanol washes, 4 min dry, elute 100 \\u00b5L, 30 s, magnet 90 s, recover. It uses a sound GEN1 height setting and adds no invented steps.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Checks '48_samples_to_odd_columns' and 'distinct_elution_wells' passed. The single-channel pipette moves each of the 48 tubes to its own odd-column well with a fresh tip.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks for volumes 40/250/250, mix 5x, 5 min incubation, magnet 4 min and magnet-on removal all passed. The code removes 540 \\u00b5L to waste with the magnet engaged.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"The code does two 500 \\u00b5L ethanol additions, each removed to waste with the magnet engaged, then a 4 min delay, then mag.disengage(). Checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"The code disengages the magnet, adds 100 \\u00b5L elution buffer and mixes 5x, waits 30 s, engages the magnet for 90 s, then moves 100 \\u00b5L to a distinct well of the elution plate, which is held at 4 C. Checks elution_100ul, elution_off_magnet_then_90s_on, recover_70_100ul_one_well_each and elution_plate_4C_before_recovery passed.\"}], \"summary\": \"The protocol passed all 17 measured run-log checks, and its code matches the paper's step order, volumes, times, magnet use and 4 C elution plate. Tip handling is clean, with one fresh tip per sample and per column, and no tips are left on the pipettes at the end, so every item scores 1.\"}]", "level": "hard", - "score": 0.6875, + "score": 1.0, "weights": { "robot_practice": 0.25, "tips_and_contamination": 0.25, @@ -214,10 +214,10 @@ "failed_votes": 0 }, "rewards": { - "reward": 0.6875, + "reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, - "judge_score": 0.6875, + "judge_score": 1.0, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, @@ -225,10 +225,10 @@ "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, - "rubric_fidelity_to_paper": 0.0, + "rubric_fidelity_to_paper": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, - "rubric_elution_recovery": 0.0 + "rubric_elution_recovery": 1.0 } } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/reward.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/reward.json index a1137c3..e89735e 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/reward.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/reward.json @@ -1 +1 @@ -{"reward": 0.6875, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 0.6875, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 0.0} +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 1.0, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/trial.json index cba673f..f0c785f 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction-hard/trial.json @@ -4,10 +4,10 @@ "agent": "claude-code", "via": "openrouter", "rewards": { - "reward": 0.6875, + "reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, - "judge_score": 0.6875, + "judge_score": 1.0, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, @@ -15,16 +15,36 @@ "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, - "rubric_fidelity_to_paper": 0.0, + "rubric_fidelity_to_paper": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, - "rubric_elution_recovery": 0.0 + "rubric_elution_recovery": 1.0 }, "exception": null, - "cost_usd": 1.2990842499999997, + "cost_usd": 0.332968, "input_tokens": 499150, "output_tokens": 15862, "started_at": "2026-10-07T13:46:00.384630Z", - "finished_at": "2026-10-07T13:57:11.856352Z" + "finished_at": "2026-10-07T13:57:11.856352Z", + "original_rewards": { + "reward": 0.6875, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 0.6875, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 0.0 + }, + "cache_tokens": 433659, + "claude_code_cost_usd": 1.2990842499999997 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction/trial.json index 5521983..f50a061 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/opentrons-rna-extraction/trial.json @@ -22,9 +22,11 @@ "rubric_elution_recovery": 1.0 }, "exception": null, - "cost_usd": 0.5695785000000001, + "cost_usd": 0.186089, "input_tokens": 295616, "output_tokens": 9857, "started_at": "2026-10-07T13:43:18.923153Z", - "finished_at": "2026-10-07T13:58:56.182538Z" + "finished_at": "2026-10-07T13:58:56.182538Z", + "cache_tokens": 265112, + "claude_code_cost_usd": 0.5695785000000001 } diff --git a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/split-200ul-two-wells/trial.json b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/split-200ul-two-wells/trial.json index 9da2cdc..89c1d31 100644 --- a/results/runs/2026-10-07-openrouter/gpt-6.1-sol/split-200ul-two-wells/trial.json +++ b/results/runs/2026-10-07-openrouter/gpt-6.1-sol/split-200ul-two-wells/trial.json @@ -19,9 +19,11 @@ "rubric_volumes_and_wells": 1.0 }, "exception": null, - "cost_usd": 0.13919275, + "cost_usd": 0.043722, "input_tokens": 55105, "output_tokens": 757, "started_at": "2026-10-07T13:57:11.879345Z", - "finished_at": "2026-10-07T13:58:29.816368Z" + "finished_at": "2026-10-07T13:58:29.816368Z", + "cache_tokens": 38978, + "claude_code_cost_usd": 0.13919275 } diff --git a/results/runs/2026-10-07-openrouter/prices.json b/results/runs/2026-10-07-openrouter/prices.json new file mode 100644 index 0000000..dc094a2 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/prices.json @@ -0,0 +1,109 @@ +{ + "deepseek/deepseek-v4-pro-0813": { + "completion": "0.00000198", + "input_cache_read": "0.000000022", + "overrides": [ + { + "completion": "0.00000198", + "input_cache_read": "0.000000022", + "prompt": "0.00000066", + "utc_days": [ + "saturday", + "sunday" + ] + }, + { + "completion": "0.00000198", + "input_cache_read": "0.000000022", + "prompt": "0.00000066", + "utc_days": [ + "monday", + "tuesday", + "wednesday", + "thursday", + "friday" + ], + "utc_end": 100, + "utc_start": 0 + }, + { + "completion": "0.00000396", + "input_cache_read": "0.000000044", + "prompt": "0.00000132", + "utc_days": [ + "monday", + "tuesday", + "wednesday", + "thursday", + "friday" + ], + "utc_end": 400, + "utc_start": 100 + }, + { + "completion": "0.00000198", + "input_cache_read": "0.000000022", + "prompt": "0.00000066", + "utc_days": [ + "monday", + "tuesday", + "wednesday", + "thursday", + "friday" + ], + "utc_end": 600, + "utc_start": 400 + }, + { + "completion": "0.00000396", + "input_cache_read": "0.000000044", + "prompt": "0.00000132", + "utc_days": [ + "monday", + "tuesday", + "wednesday", + "thursday", + "friday" + ], + "utc_end": 1000, + "utc_start": 600 + }, + { + "completion": "0.00000198", + "input_cache_read": "0.000000022", + "prompt": "0.00000066", + "utc_days": [ + "monday", + "tuesday", + "wednesday", + "thursday", + "friday" + ], + "utc_end": 0, + "utc_start": 1000 + } + ], + "prompt": "0.00000066" + }, + "openai/gpt-6.1-sol": { + "completion": "0.00001", + "input_cache_read": "0.0000001", + "input_cache_write": "0.0000025", + "overrides": [ + { + "completion": "0.000015", + "input_cache_read": "0.0000002", + "input_cache_write": "0.000005", + "min_prompt_tokens": 272000, + "prompt": "0.000004" + } + ], + "prompt": "0.000002", + "web_search": "0.01" + }, + "qwen/qwen3.8-2.4t-a95b": { + "completion": "0.000006", + "input_cache_read": "0.00000025", + "prompt": "0.000002" + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/grader.json new file mode 100644 index 0000000..e62de75 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/grader.json @@ -0,0 +1,137 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 61 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 is valid, 100 uL is handled by the p300, and the labware use is sensible. All measured checks passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Each transfer uses new_tip='always', which picks up and drops a tip every time. The tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "The protocol does the single 100 uL transfer from reservoir A1 to plate A1:A12 in order, with the specified labware and labels. A fresh tip per well is allowed, and the comment is accurate.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "deck_and_hardware", + "score": 1, + "evidence": "The reservoir is in slot 1 and the plate in slot 2, both with the specified labels. The p20 is on the left with the 20 uL rack in slot 10, and the p300 is on the right with the 300 uL rack in slot 11.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "volumes_and_wells", + "score": 1, + "evidence": "The code transfers 100 uL with p300 from reservoir A1 to each of plate A1-A12 and touches no other wells. The end_state:plate check passed.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "deck_and_hardware": 1.0, + "volumes_and_wells": 1.0 + }, + "summary": "The protocol transfers 100 uL from reservoir A1 to plate wells A1-A12 with the p300, using a fresh tip for each well. All measured checks passed and I found no deviations from the task.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "deck_and_hardware": 0.125, + "volumes_and_wells": 0.125 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/protocol.py new file mode 100644 index 0000000..d482230 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/protocol.py @@ -0,0 +1,40 @@ +metadata = { + 'protocolName': 'Reservoir to 96-well plate A1:A12 - 100 uL', + 'description': 'Transfer 100 uL from a 1-well reservoir (A1) to plate wells A1 through A12.', + 'apiLevel': '2.15', +} + + +def run(protocol): + # Labware + reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 1, label='reservoir') + plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='plate') + + # Pipettes and tips + tips_20 = protocol.load_labware( + 'opentrons_96_tiprack_20ul', 10, label='20 uL tips') + tips_300 = protocol.load_labware( + 'opentrons_96_tiprack_300ul', 11, label='300 uL tips') + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips_300]) + + # 100 uL is within the p300 range (20-300 uL); the p20 cannot handle it. + source = reservoir.wells_by_name()['A1'] + destinations = [plate.wells_by_name()[f'A{i}'] for i in range(1, 13)] + + protocol.comment('Transferring 100 uL from reservoir A1 to plate A1:A12') + + for well in destinations: + p300.transfer(100, source, well, new_tip='always') + + # Ensure no tip is left on the pipette at the end (transfer drops it, + # but be explicit for safety). + if p300.has_tip: + p300.drop_tip() + if p20.has_tip: + p20.drop_tip() diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/reward.json new file mode 100644 index 0000000..5b16841 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/trial.json new file mode 100644 index 0000000..13789b1 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/a1-a12-100ul/trial.json @@ -0,0 +1,29 @@ +{ + "task": "a1-a12-100ul", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + }, + "exception": null, + "cost_usd": 0.03441, + "input_tokens": 51291, + "output_tokens": 1182, + "started_at": "2026-10-07T17:52:15.318212Z", + "finished_at": "2026-10-07T17:53:20.229231Z", + "cache_tokens": 43008, + "claude_code_cost_usd": 0.092469 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/grader.json new file mode 100644 index 0000000..1458b5f --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/grader.json @@ -0,0 +1,168 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 7316 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:sample_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:elution_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, all volumes (40\u2013200 \u00b5L) handled by p300 within range, labware loaded with the required labels, and no overdispense or empty-well check failed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Every transfer uses new_tip='always' (a fresh tip per well), the checks tip_before_aspirate, no_cross_contamination and tip_dropped_at_end pass, and reset_tipracks keeps tips available.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All 13 steps appear in order with the correct volumes, mix counts and comments for the non-simulated steps, and nothing extra is added.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding", + "score": 1, + "evidence": "The loop does p300.transfer(40, beads, well, mix_after=(10,40)) for all 96 wells, then comments the 5 min incubation before the magnet.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "supernatant_and_washes", + "score": 1, + "evidence": "The magnet comment comes first, then 90 \u00b5L is removed per well to waste, then two 200 \u00b5L ethanol adds each followed by a 200 \u00b5L removal to waste.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "drying_and_elution", + "score": 1, + "evidence": "Comments cover the 5 min air dry, magnet disengage and 2 min incubation with magnet re-engage; 50 \u00b5L water is added with mix_after=(10,50).", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "recovery", + "score": 1, + "evidence": "p300.transfer(45, sample_wells, elution_wells, new_tip='always') pairs wells in order, and the end_state:elution_plate check passes.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "binding": 1.0, + "supernatant_and_washes": 1.0, + "drying_and_elution": 1.0, + "recovery": 1.0 + }, + "summary": "The protocol follows the task steps in order with correct volumes, a fresh tip for every well, and all measured checks passing. The non-simulated steps are recorded as protocol.comment calls, as the task requires.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "binding": 0.0625, + "supernatant_and_washes": 0.0625, + "drying_and_elution": 0.0625, + "recovery": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_binding": 1.0, + "rubric_supernatant_and_washes": 1.0, + "rubric_drying_and_elution": 1.0, + "rubric_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/protocol.py new file mode 100644 index 0000000..5b9ab6b --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/protocol.py @@ -0,0 +1,102 @@ +metadata = {'apiLevel': '2.13'} + + +def run(protocol): + # Labware + sample_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='sample_plate') + beads_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 2, label='beads_reservoir') + ethanol_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='ethanol_reservoir') + water_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 4, label='water_reservoir') + waste = protocol.load_labware( + 'nest_1_reservoir_195ml', 5, label='waste') + elution_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='elution_plate') + + # Tip racks + tiprack_20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tiprack_300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + # Pipettes + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tiprack_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tiprack_300]) + + sample_wells = sample_plate.wells() # A1:H12, row-major + elution_wells = elution_plate.wells() # A1:H12, row-major (pairs in order) + waste_well = [waste['A1']] * len(sample_wells) + ethanol_source = [ethanol_reservoir['A1']] * len(sample_wells) + beads_source = beads_reservoir['A1'] + water_source = water_reservoir['A1'] + + # Step 1: add 40 uL AMPure XP beads to each sample well, mix 10x + protocol.comment('Step 1: Adding 40 uL AMPure XP beads to sample plate') + for well in sample_wells: + p300.transfer(40, beads_source, well, + mix_after=(10, 40), new_tip='always') + p300.reset_tipracks() + + # Step 2: incubate + protocol.comment( + 'Step 2: Incubate sample_plate 5 min at room temperature ' + '(beads bind DNA)') + + # Step 3: engage magnet + protocol.comment( + 'Step 3: Engage magnetic module; wait 5 min until solution clears') + + # Step 4: remove 90 uL supernatant from each sample well to waste + protocol.comment('Step 4: Removing 90 uL supernatant to waste') + p300.transfer(90, sample_wells, waste_well, new_tip='always') + p300.reset_tipracks() + + # Step 5: first 200 uL 80% ethanol wash + protocol.comment('Step 5: Adding 200 uL 80% ethanol (wash 1)') + p300.transfer(200, ethanol_source, sample_wells, new_tip='always') + p300.reset_tipracks() + + # Step 6: remove 200 uL ethanol waste 1 + protocol.comment('Step 6: Removing 200 uL ethanol waste 1') + p300.transfer(200, sample_wells, waste_well, new_tip='always') + p300.reset_tipracks() + + # Step 7: second 200 uL 80% ethanol wash + protocol.comment('Step 7: Adding 200 uL 80% ethanol (wash 2)') + p300.transfer(200, ethanol_source, sample_wells, new_tip='always') + p300.reset_tipracks() + + # Step 8: remove 200 uL ethanol waste 2 + protocol.comment('Step 8: Removing 200 uL ethanol waste 2') + p300.transfer(200, sample_wells, waste_well, new_tip='always') + p300.reset_tipracks() + + # Step 9: air dry + protocol.comment( + 'Step 9: Air dry beads 5 min at room temperature (magnet engaged); ' + 'beads should appear matte not shiny') + + # Step 10: disengage magnet + protocol.comment('Step 10: Disengage magnetic module') + + # Step 11: elute in 50 uL nuclease-free water, mix 10x + protocol.comment('Step 11: Adding 50 uL nuclease-free water to elute') + for well in sample_wells: + p300.transfer(50, water_source, well, + mix_after=(10, 50), new_tip='always') + p300.reset_tipracks() + + # Step 12: incubate and re-engage magnet + protocol.comment( + 'Step 12: Incubate sample_plate 2 min at room temperature; ' + 'then re-engage magnetic module 5 min') + + # Step 13: transfer 45 uL cleaned DNA to elution plate + protocol.comment('Step 13: Transferring 45 uL cleaned DNA to elution plate') + p300.transfer(45, sample_wells, elution_wells, new_tip='always') + p300.reset_tipracks() + + protocol.comment('AMPure XP cleanup complete.') diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/reward.json new file mode 100644 index 0000000..48d40a0 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_binding": 1.0, "rubric_supernatant_and_washes": 1.0, "rubric_drying_and_elution": 1.0, "rubric_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/trial.json new file mode 100644 index 0000000..6006f9a --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ampure-bead-cleanup/trial.json @@ -0,0 +1,31 @@ +{ + "task": "ampure-bead-cleanup", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_binding": 1.0, + "rubric_supernatant_and_washes": 1.0, + "rubric_drying_and_elution": 1.0, + "rubric_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.153528, + "input_tokens": 216193, + "output_tokens": 10345, + "started_at": "2026-10-07T18:15:30.981264Z", + "finished_at": "2026-10-07T18:20:32.307513Z", + "cache_tokens": 194816, + "claude_code_cost_usd": 0.46291800000000005 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/grader.json new file mode 100644 index 0000000..4d7baf0 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/grader.json @@ -0,0 +1,162 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 2309 + }, + "not_applicable": [ + "end_state:pcr_plate (the source paper does not determine the exact volumes; composition checked instead)" + ], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:colony_plate", + "pass": true, + "detail": "" + }, + { + "name": "composition:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "Valid apiLevel 2.13 and metadata; all volumes (1-5 uL, 8 uL mix) are handled by the p20 within range, and the 10 uL wells are never overfilled. All measured checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Every step does pick_up_tip/drop_tip per well, including a fresh tip for each primer pair and colony. The tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "The protocol adds steps the paper does not have (mix and touch_tip). It also invents assumptions about primer Tm and concentration, and it describes an 'ice-chilled deck' that the code does not provide. Cycling uses a combined 72 C anneal/extend with a '4-10 C' hold instead of the paper-based adjusted program.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "reaction_setup", + "score": 1, + "evidence": "10 uL reaction: 5 uL Q5 2x master mix is dispensed first, then 4 uL primer pair, then 1 uL colony template.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "sample_mapping", + "score": 1, + "evidence": "colony_plate.wells(), primer_plate.wells() and pcr_plate.wells() are zipped in order, so all 96 wells map A1 to A1 with none skipped or reused.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "thermocycling", + "score": 0, + "evidence": "Sealing and cycling are recorded only in a final comment. The program is a two-step 98/72 C cycle with no separate annealing step. It rests on an invented assumption (Tm >= 69 C) and ends with a '4-10 C' hold rather than 4 C.", + "votes": [ + 0, + 1, + 0 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "reaction_setup": 1.0, + "sample_mapping": 1.0, + "thermocycling": 0.0 + }, + "summary": "The liquid handling is sound. It uses fresh tips throughout, builds correct 10 uL reactions in the right order, and maps wells 1:1 across all 96. It loses the fidelity and thermocycling items because of invented assumptions (primer Tm, primer concentration, ice-chilled deck), extra mixing and touch-tip steps, and a non-standard two-step cycling program with a 4-10 C hold.", + "level": "hard", + "score": 0.6667, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "reaction_setup": 0.0833, + "sample_mapping": 0.0833, + "thermocycling": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 0.6667, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.6667, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 0.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/protocol.py new file mode 100644 index 0000000..a27d6d4 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/protocol.py @@ -0,0 +1,172 @@ +""" +Slowpoke-style colony PCR screening of 96 transformants on the OT-2. + +Based on the Slowpoke paper (ACS Synth. Biol., doi:10.1021/acssynbio.5c00629), +"Automated Colony PCR" (OT-2 workflow): a 10 uL total reaction is assembled by +dispensing the master mix into each reaction well first, followed by 1 uL of +colony template; the paper's mix (9 uL of water + primers + 2x PCR master mix, +Phire Plant Direct) is then thermocycled per the manufacturer's protocol, with +annealing temperature and extension time adjusted for the fragment/primers. + +Adaptation to the fixed deck loaded here (choices stated per the task brief): + * Polymerase: Q5 Hot Start 2x master mix (NEB) instead of Phire Plant + Direct. Q5 is a 2x mix, so it is used at half the final reaction volume: + 5 uL per 10 uL reaction (final 1x, per the manufacturer). + * No water reservoir and no bulk-primer tube are on this deck; instead each + well of `primer_plate` holds the (premixed forward + reverse) screening + primer pair for the matching colony. The water component of the paper's + 9 uL master mix is therefore replaced by primer solution: we transfer + 4 uL of primer pair per reaction. ASSUMPTION: the primer pairs in the + plate are dilute enough that 4 uL into 10 uL gives the NEB-recommended + 0.5-1 uM final primer concentration each (i.e. ~1.25-2.5 uM each primer + in the plate). If the plate holds standard 10 uM working stocks, the + operator must dilute them before loading, since the deck has no water. + * Colony template: 1 uL per reaction, exactly as the OT-2 workflow in the + paper (not the 2 uL Flex variant). ASSUMPTION: `colony_plate` holds + picked colonies resuspended in liquid (water/TE or growth medium), as + prepared manually in the Slowpoke workflow. + * Reaction layout (per well of `pcr_plate`, 10 uL total): + 5 uL Q5 Hot Start 2x master mix + 4 uL primer pair (from the matching well of `primer_plate`) + 1 uL colony template (from the matching well of `colony_plate`) + * Well mapping: well i of `colony_plate` <-> well i of `primer_plate` <-> + well i of `pcr_plate` (A1:H12, all 96 wells screened). + * All volumes (1-5 uL) fall in the p20_single_gen2 range (1-20 uL), so the + p20 does all liquid handling; the p300 (min 20 uL) is loaded but unused. + A fresh tip is used for every aspiration (tips are unlimited; racks are + reset between phases). + * No thermocycler module is on this deck, so after reaction setup the plate + is sealed and moved to a benchtop thermocycler (the paper allows benchtop + cyclers alongside/instead of the Opentrons module). Cycling follows the + Q5 Hot Start manufacturer's two-step protocol, adjusted as the paper + describes for amplicon size and primer Tm (assumed <=1 kb amplicon, + primers Tm >= 69 C): + 98 C 30 s initial denaturation / hot-start activation + 30 cycles of: + 98 C 10 s denaturation + 72 C 30 s combined anneal/extend (~30 s/kb for <=1 kb) + 72 C 2 min final extension + 4-10 C hold +""" + +metadata = { + 'apiLevel': '2.13', + 'protocolName': 'Slowpoke OT-2 Colony PCR (Q5 2x, 96 colonies)', + 'description': 'Set up 96 x 10 uL colony PCR reactions from a colony ' + 'template plate, a per-colony primer plate and a Q5 Hot ' + 'Start 2x master mix reservoir, following the Slowpoke ' + 'OT-2 colony PCR workflow.', + 'author': 'Claude', +} + +# Per-reaction volumes (uL); 10 uL total, matching the paper's OT-2 workflow. +MM_VOL = 5.0 # Q5 Hot Start 2x master mix -> 1x final +PRIMER_VOL = 4.0 # premixed primer pair (replaces the paper's mix water) +COLONY_VOL = 1.0 # colony template, as in the paper's OT-2 workflow +MIX_VOL = 8.0 # pipette-mix volume after template addition (well holds 10 uL) + + +def run(protocol): + # ---------------- Deck (fixed) ---------------- + colony_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='colony_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='pcr_plate') + mm_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='master_mix_reservoir') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='primer_plate') + + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + # p300 is part of the fixed setup but every volume here (1-5 uL) is below + # its 20 uL minimum, so the p20 does all of the liquid handling. + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + mm_source = mm_reservoir.wells()[0] + + # Consistent well-to-well mapping: A1:H12 on all three plates. + colony_wells = colony_plate.wells() + primer_wells = primer_plate.wells() + pcr_wells = pcr_plate.wells() + n_rxns = len(pcr_wells) # 96 + + protocol.comment( + '=== Slowpoke colony PCR: %d x 10 uL reactions ===' + 'Setup: %s uL Q5 2x master mix + %s uL primer pair + %s uL colony ' + 'template. Ensure colony_plate wells contain picked colonies ' + 'resuspended in liquid, primer_plate wells contain the premixed ' + 'primer pair for the matching colony, and the reservoir holds Q5 Hot ' + 'Start 2x master mix on ice-chilled deck as per usual practice.' + % (n_rxns, MM_VOL, PRIMER_VOL, COLONY_VOL)) + + # Gentle handling of the enzyme mix and primers. + p20.flow_rate.aspirate = 30 # uL/s (below default ~57) for accuracy + p20.flow_rate.dispense = 30 + + # ---------------- Step 1: dispense master mix ---------------- + # Paper: master mix is dispensed into every reaction well first, colony + # template added last. Uses 96 x 5 uL = 480 uL of the 2x Q5 mix + # (reservoir is stated to hold plenty). + protocol.comment('Step 1/3: dispensing %s uL Q5 Hot Start 2x master mix ' + 'into all %d wells of pcr_plate.' % (MM_VOL, n_rxns)) + for dest in pcr_wells: + p20.pick_up_tip() + p20.aspirate(MM_VOL, mm_source.bottom(z=5)) + p20.dispense(MM_VOL, dest.bottom(z=1)) + p20.blow_out(dest.top(z=-2)) + p20.drop_tip() + # Rack (96 tips) exhausted; tips are unlimited -> reset for next phase. + p20.reset_tipracks() + + # ---------------- Step 2: add per-colony primer pairs ---------------- + # 4 uL of each colony's premixed primer pair; fresh tip per well (pairs + # differ well-to-well, so no tip may be reused). + protocol.comment('Step 2/3: adding %s uL of each colony\'s primer pair ' + 'from primer_plate to pcr_plate (fresh tip per well).' + % PRIMER_VOL) + for src, dest in zip(primer_wells, pcr_wells): + p20.pick_up_tip() + p20.aspirate(PRIMER_VOL, src.bottom(z=1)) + p20.dispense(PRIMER_VOL, dest.bottom(z=1)) + p20.blow_out(dest.top(z=-2)) + p20.touch_tip() + p20.drop_tip() + p20.reset_tipracks() + + # ---------------- Step 3: add colony template ---------------- + # 1 uL per reaction, as in the paper's OT-2 workflow. Fresh tip per well + # to avoid cross-contaminating colonies. + protocol.comment('Step 3/3: adding %s uL colony template from colony_plate ' + 'to each reaction.' % COLONY_VOL) + for src, dest in zip(colony_wells, pcr_wells): + p20.pick_up_tip() + p20.aspirate(COLONY_VOL, src.bottom(z=0.5)) + p20.dispense(COLONY_VOL, dest.bottom(z=1)) + # Mix the 10 uL reaction gently with the same tip, then blow out. + p20.mix(3, MIX_VOL, dest.bottom(z=1)) + p20.blow_out(dest.top(z=-2)) + p20.touch_tip() + p20.drop_tip() + + # ---------------- Manual steps the robot cannot do ---------------- + protocol.comment( + 'Reaction setup complete (96 x 10 uL). MANUAL STEPS: ' + '1) Seal pcr_plate with an optical/PCR seal and briefly centrifuge. ' + '2) Transfer the sealed plate to a benchtop thermocycler (no ' + 'thermocycler module on this deck; the Slowpoke workflow also uses ' + 'benchtop cyclers). ' + '3) Run Q5 Hot Start two-step cycling (per NEB manufacturer protocol, ' + 'adjusted for fragment size/primer Tm as in the paper; assumes ' + '<=1 kb amplicon, primer Tm >= 69 C): ' + '98 C 30 s initial denaturation; ' + '30 cycles of [98 C 10 s; 72 C 30 s combined anneal/extend]; ' + '72 C 2 min final extension; hold 4-10 C. ' + 'For longer amplicons extend the 72 C step (~30 s/kb); for lower-Tm ' + 'primers use a 3-step program with an appropriate annealing step. ' + '4) Analyse products by agarose gel electrophoresis to score ' + 'positive colonies by expected amplicon size.') diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/reward.json new file mode 100644 index 0000000..e4df621 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.6667, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 0.6667, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_reaction_setup": 1.0, "rubric_sample_mapping": 1.0, "rubric_thermocycling": 0.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/trial.json new file mode 100644 index 0000000..eed103d --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening-hard/trial.json @@ -0,0 +1,30 @@ +{ + "task": "colony-pcr-screening-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.6667, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.6667, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_reaction_setup": 1.0, + "rubric_sample_mapping": 1.0, + "rubric_thermocycling": 0.0 + }, + "exception": null, + "cost_usd": 0.167832, + "input_tokens": 193193, + "output_tokens": 12817, + "started_at": "2026-10-07T17:47:15.084953Z", + "finished_at": "2026-10-07T17:52:54.097537Z", + "cache_tokens": 168832, + "claude_code_cost_usd": 0.5266460000000001 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/grader.json new file mode 100644 index 0000000..534a6f4 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/grader.json @@ -0,0 +1,160 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1828 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:colony_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, all volumes (18, 1, 10 \u00b5L mix) handled by p20 within its range, labware loaded with the specified labels, and all simulator checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Every well uses its own pick_up_tip()/drop_tip() pair in all three loops; tip_before_aspirate, no_cross_contamination and tip_dropped_at_end checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "The steps run in the specified order with the specified quantities (18 \u00b5L, then 1 \u00b5L colony with mix(3, 10) after dispensing, then 1 \u00b5L primer), and the final comment records seal and thermocycling.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "master_mix", + "score": 1, + "evidence": "The loop aspirates 18 \u00b5L from master_mix_reservoir A1 and dispenses it into each of the 96 pcr_plate wells; the pcr_plate end_state check passes.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "template_and_primers", + "score": 1, + "evidence": "The loops move 1 \u00b5L from colony_plate[name] and 1 \u00b5L from primer_plate[name] into the matching pcr_plate well with p20; the end-state checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "thermocycling", + "score": 1, + "evidence": "The final protocol.comment records sealing and the full program: 98C 30 s, 30 cycles of 98C 10 s, 60C 30 s and 72C 30 s, then 72C 2 min and a 4C hold.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "master_mix": 1.0, + "template_and_primers": 1.0, + "thermocycling": 1.0 + }, + "summary": "The protocol simulates cleanly and passes every measured check. It uses a fresh p20 tip for each well, follows the quantities and order in the task, and records the seal and thermocycling program in a comment.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "master_mix": 0.0833, + "template_and_primers": 0.0833, + "thermocycling": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_master_mix": 1.0, + "rubric_template_and_primers": 1.0, + "rubric_thermocycling": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/protocol.py new file mode 100644 index 0000000..b422d17 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/protocol.py @@ -0,0 +1,91 @@ +"""Colony PCR screening with Q5 Hot Start master mix (Slowpoke workflow). + +OT-2 protocol: add 18 uL Q5 Hot Start master mix to each well of a PCR +plate, then 1 uL colony template (mixed), then 1 uL primer mix per well. +Based on Slowpoke: Golden Gate cloning and colony PCR on OT-2/Flex +(ACS Synth. Biol., doi:10.1021/acssynbio.5c00629). +""" + +from opentrons import protocol_api + +metadata = { + 'protocolName': 'Colony PCR screening with Q5 Hot Start master mix', + 'author': 'Slowpoke workflow', + 'description': 'Screen transformant colonies by PCR on the OT-2.', + 'apiLevel': '2.13', +} + + +def run(protocol: protocol_api.ProtocolContext): + # ------------------------------------------------------------------ + # Labware (fixed deck layout) + # ------------------------------------------------------------------ + colony_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='colony_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='pcr_plate') + master_mix_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='master_mix_reservoir') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='primer_plate') + + tips_20 = protocol.load_labware( + 'opentrons_96_tiprack_20ul', 10) + tips_300 = protocol.load_labware( + 'opentrons_96_tiprack_300ul', 11) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips_300]) + + # All 96 wells, A1..H12, in order + wells = [f'{row}{col}' for row in 'ABCDEFGH' for col in range(1, 13)] + + # ------------------------------------------------------------------ + # Step 1: 18 uL Q5 master mix 2x from reservoir A1 -> pcr_plate A1:H12 + # (18 uL is within the p20 range: 1-20 uL) + # ------------------------------------------------------------------ + protocol.comment('Step 1: Dispensing 18 uL Q5 Hot Start master mix ' + 'into pcr_plate wells A1:H12.') + p20.reset_tipracks() + mm_source = master_mix_reservoir['A1'] + for name in wells: + p20.pick_up_tip() + p20.aspirate(18, mm_source) + p20.dispense(18, pcr_plate[name]) + p20.drop_tip() + + # ------------------------------------------------------------------ + # Step 2: 1 uL colony template colony_plate A1:H12 -> pcr_plate A1:H12, + # mix 3x after dispensing + # ------------------------------------------------------------------ + protocol.comment('Step 2: Adding 1 uL colony template to pcr_plate ' + 'wells A1:H12, mixing 3 times after dispensing.') + p20.reset_tipracks() + for name in wells: + p20.pick_up_tip() + p20.aspirate(1, colony_plate[name]) + p20.dispense(1, pcr_plate[name]) + # mix 3 times (well holds 19 uL at this point) + p20.mix(3, 10, pcr_plate[name]) + p20.drop_tip() + + # ------------------------------------------------------------------ + # Step 3: 1 uL primer mix primer_plate A1:H12 -> pcr_plate A1:H12 + # ------------------------------------------------------------------ + protocol.comment('Step 3: Adding 1 uL primer mix to pcr_plate ' + 'wells A1:H12.') + p20.reset_tipracks() + for name in wells: + p20.pick_up_tip() + p20.aspirate(1, primer_plate[name]) + p20.dispense(1, pcr_plate[name]) + p20.drop_tip() + + # ------------------------------------------------------------------ + # Step 4: seal and thermocycle (manual, not simulated) + # ------------------------------------------------------------------ + protocol.comment('Step 4: Seal pcr_plate and thermocycle: ' + '98C for 30 s; 30 cycles of [98C 10 s, 60C 30 s, ' + '72C 30 s]; 72C for 2 min; hold at 4C.') diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/reward.json new file mode 100644 index 0000000..12f62ba --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_master_mix": 1.0, "rubric_template_and_primers": 1.0, "rubric_thermocycling": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/trial.json new file mode 100644 index 0000000..7d2bfde --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/colony-pcr-screening/trial.json @@ -0,0 +1,30 @@ +{ + "task": "colony-pcr-screening", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_master_mix": 1.0, + "rubric_template_and_primers": 1.0, + "rubric_thermocycling": 1.0 + }, + "exception": null, + "cost_usd": 0.084508, + "input_tokens": 133465, + "output_tokens": 4503, + "started_at": "2026-10-07T18:20:32.381252Z", + "finished_at": "2026-10-07T18:23:18.337231Z", + "cache_tokens": 119680, + "claude_code_cost_usd": 0.24134 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/grader.json new file mode 100644 index 0000000..e780f43 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/grader.json @@ -0,0 +1,155 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 203 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plasmid_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:transformation_plate", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "requirements apiLevel 2.15 is valid; 1 \u00b5L and 5 \u00b5L mixes use p20, 50 \u00b5L and 40 \u00b5L mix use p300; the 61 \u00b5L final volume is well within the 200 \u00b5L wells; the simulator ran clean.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Each of the 8 DNA transfers and 8 SOC additions uses pick_up_tip and drop_tip, giving a fresh tip per well; the tip_before_aspirate, tip_dropped_at_end and no_cross_contamination checks all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "The protocol adds steps the paper does not describe: p20.mix(3,5) with blow_out after DNA addition, p300.mix(3,40) after SOC, and no lid heating, so it is not a fully faithful implementation.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "dna_addition", + "score": 1, + "evidence": "tc.set_block_temperature(4) with the lid open, then 1 \u00b5L aspirated from plasmid well n and dispensed into transformation well n for A1\u2013H1.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "heat_shock", + "score": 1, + "evidence": "tc.close_lid(), then set_block_temperature(4, hold_time_minutes=30), then set_block_temperature(42, hold_time_seconds=30), then a return to 4 \u00b0C.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "soc_recovery", + "score": 1, + "evidence": "SOC_VOL=50 is dispensed into each cell well with p300 after the heat shock, then tc.close_lid() and set_block_temperature(37, hold_time_minutes=60).", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "dna_addition": 1.0, + "heat_shock": 1.0, + "soc_recovery": 1.0 + }, + "summary": "The protocol follows the LV volumes and the 4 \u00b0C, 42 \u00b0C and 37 \u00b0C steps on the thermocycler, with fresh tips throughout, and all measured checks passed. It fails the all-or-nothing fidelity item because it adds mixing steps (and omits lid heating) that the paper does not describe.", + "level": "hard", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "dna_addition": 0.0833, + "heat_shock": 0.0833, + "soc_recovery": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/protocol.py new file mode 100644 index 0000000..2bdda29 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/protocol.py @@ -0,0 +1,137 @@ +"""APEX Protocol 1 - Automated heat shock transformation on the OT-2 thermocycler. + +Implements the heat shock transformation of Kasprzyk, Herrera & Stracquadanio, +"APEX: Automated Protein EXpression in Escherichia coli" (bioRxiv 2024, +doi:10.1101/2024.08.13.607171), Protocol 1, for 8 plasmids (pEX01-pEX08). + +Parameters taken from the paper: + * Volumes: the Low Volume (LV) configuration, which the paper selected for + all subsequent automated experiments - 1 uL plasmid DNA, 10 uL competent + cells, 50 uL SOC medium (Results > Optimisation of transformation volume; + Fig. 4 caption). + * Temperatures/times (Protocol 1 defaults): incubate DNA + cells at 4 degC + for 30 min, heat shock at 42 degC for 30 s, recover with SOC at 37 degC + for 1 h, all on the thermocycler module for rapid temperature changes. + +Choices where the paper leaves details open are flagged with comments below. +Steps after recovery (spotting on agar, Protocol 2) are out of scope. +""" + +metadata = { + 'protocolName': 'APEX Protocol 1 - Heat shock transformation (LV, 8 plasmids)', + 'author': 'generated from doi:10.1101/2024.08.13.607171', + 'description': 'Automated heat shock transformation of 8 plasmids ' + '(pEX01-pEX08) into DH5alpha on the OT-2 thermocycler ' + 'using the APEX Low Volume configuration.', +} + +requirements = {'robotType': 'OT-2', 'apiLevel': '2.15'} + + +def run(protocol): + # ------------------------------------------------------------------ + # Labware and instruments (fixed deck layout) + # ------------------------------------------------------------------ + plasmid_plate = protocol.load_labware( + 'biorad_96_wellplate_200ul_pcr', 1, label='plasmid_plate') + soc_reservoir = protocol.load_labware( + 'nest_12_reservoir_15ml', 2, label='soc_reservoir') + + tc = protocol.load_module('thermocycler') + transformation_plate = tc.load_labware( + 'biorad_96_wellplate_200ul_pcr', label='transformation_plate') + + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 4) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 5) + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + # ------------------------------------------------------------------ + # Volumes - APEX Low Volume (LV) configuration selected in the paper + # ------------------------------------------------------------------ + DNA_VOL = 1.0 # uL plasmid (LV); cells are 10 uL, pre-loaded by operator + SOC_VOL = 50.0 # uL SOC medium (LV) + + # Transformation n uses plasmid well n and cell well n (A1..H1, col 1) + well_names = ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1', 'H1'] + dna_wells = [plasmid_plate[w] for w in well_names] + cell_wells = [transformation_plate[w] for w in well_names] + soc = soc_reservoir['A1'] + + # ------------------------------------------------------------------ + # Step 1: add plasmid DNA to the pre-chilled competent cells on the + # thermocycler block (kept at 4 degC, as in the paper's ice-cold + # CaCl2 pre-incubation). One fresh tip per plasmid to avoid + # cross-contamination. + # ------------------------------------------------------------------ + tc.open_lid() + # Ensure the block is at the 4 degC incubation temperature (the cells + # were loaded by the operator onto the pre-chilled block). + tc.set_block_temperature(4) + + for src, dst in zip(dna_wells, cell_wells): + p20.pick_up_tip() + p20.aspirate(DNA_VOL, src) + p20.dispense(DNA_VOL, dst) + # The paper does not specify mixing after DNA addition; mix 3x 5 uL + # (well holds 11 uL total) to homogenise the 1 uL DNA in the cells, + # then blow out any remaining droplets. + p20.mix(3, 5, dst) + p20.blow_out(dst.top()) + p20.drop_tip() + + # Reset the 20 uL rack so a full rack is available if this protocol is + # re-run/appended; tips are unlimited per the task setup. + p20.reset_tipracks() + + # ------------------------------------------------------------------ + # Step 2: 4 degC for 30 min incubation, lid closed to limit evaporation. + # (The paper does not mention heating the lid, so no lid temperature is + # set; the closed lid is the sound choice for these small volumes.) + # ------------------------------------------------------------------ + tc.close_lid() + tc.set_block_temperature(4, hold_time_minutes=30) + + # ------------------------------------------------------------------ + # Step 3: heat shock - 42 degC for 30 s. Default (maximum) ramp rate is + # used, since the paper stresses the need for rapid temperature shifts. + # ------------------------------------------------------------------ + tc.set_block_temperature(42, hold_time_seconds=30) + + # Cool the block back to 4 degC before opening the lid (standard heat + # shock practice: return cells to ice-cold immediately after the shock; + # the paper does not state this explicitly, so it is a sound choice). + tc.set_block_temperature(4) + + # ------------------------------------------------------------------ + # Step 4: add 50 uL SOC (LV volume) to each transformation and mix. + # One tip per well; fresh tips for the cell/SOC mixture. + # ------------------------------------------------------------------ + tc.open_lid() + for dst in cell_wells: + p300.pick_up_tip() + p300.aspirate(SOC_VOL, soc) + p300.dispense(SOC_VOL, dst) + # Mix below the total well volume (10 + 1 + 50 = 61 uL) to avoid + # spilling; the paper does not specify a mixing volume. + p300.mix(3, 40, dst) + p300.blow_out(dst.top()) + p300.drop_tip() + p300.reset_tipracks() + + # ------------------------------------------------------------------ + # Step 5: recovery - 37 degC for 1 h with SOC, lid closed. + # ------------------------------------------------------------------ + tc.close_lid() + tc.set_block_temperature(37, hold_time_minutes=60) + + # ------------------------------------------------------------------ + # End of Protocol 1: open the lid and deactivate the block. The + # recovered outgrowths are handled by Protocol 2 (spotting), which is + # out of scope here. + # ------------------------------------------------------------------ + tc.open_lid() + tc.deactivate_block() diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/reward.json new file mode 100644 index 0000000..cc1f6c6 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.75, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 0.75, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/trial.json new file mode 100644 index 0000000..81d263c --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation-hard/trial.json @@ -0,0 +1,30 @@ +{ + "task": "ecoli-heat-shock-transformation-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.75, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 0.75, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.173724, + "input_tokens": 299108, + "output_tokens": 8398, + "started_at": "2026-10-07T18:11:31.833572Z", + "finished_at": "2026-10-07T18:15:30.920527Z", + "cache_tokens": 271360, + "claude_code_cost_usd": 0.4843700000000001 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/grader.json new file mode 100644 index 0000000..5670592 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/grader.json @@ -0,0 +1,155 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 12 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plasmid_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:cells_rack", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, p20 used for 2 \u00b5L and p300 for 250 \u00b5L, correct labware, all checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "transfer with new_tip='always' gives fresh tips for DNA and SOC; tip_before_aspirate, tip_dropped_at_end and no_cross_contamination pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All four steps in order with correct quantities, labware labels and accurate comments; no extra actions.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dna_addition", + "score": 1, + "evidence": "p20.transfer(2, plasmid_plate['A1'], tubes A1) with the 20 \u00b5L pipette.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "heat_shock", + "score": 1, + "evidence": "Comment records 42\u00b0C 45 s then ice 2 min after DNA addition and before SOC.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "soc_recovery", + "score": 1, + "evidence": "p300.transfer 250 \u00b5L from reservoir A1 to tube A1 after heat shock, followed by the 37\u00b0C 60 min 250 rpm incubation comment.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "dna_addition": 1.0, + "heat_shock": 1.0, + "soc_recovery": 1.0 + }, + "summary": "The protocol follows the task steps in order with correct pipettes, volumes, and fresh tips. All measured checks passed and the comments are accurate.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "dna_addition": 0.0833, + "heat_shock": 0.0833, + "soc_recovery": 0.0833 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/protocol.py new file mode 100644 index 0000000..9f1913e --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/protocol.py @@ -0,0 +1,54 @@ +from opentrons import protocol_api + +metadata = { + 'apiLevel': '2.13', + 'protocolName': 'E. coli heat shock transformation with SOC recovery', +} + + +def run(protocol: protocol_api.ProtocolContext): + # Deck setup + plasmid_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 1, label='plasmid_plate') + tubes_1_5ml_1 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + soc_reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 3, label='soc_reservoir') + + tiprack_20 = protocol.load_labware( + 'opentrons_96_tiprack_20ul', 10, label='tiprack_20ul') + tiprack_300 = protocol.load_labware( + 'opentrons_96_tiprack_300ul', 11, label='tiprack_300ul') + + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tiprack_20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tiprack_300]) + + cells_tube = tubes_1_5ml_1['A1'] + + # Step 1: add 2 uL plasmid DNA to competent cells + p20.transfer( + 2, + plasmid_plate['A1'], + cells_tube, + new_tip='always', + ) + + # Step 2: heat shock (not simulated) + protocol.comment( + 'Heat shock cells_rack 42°C 45 s, then transfer immediately to ' + 'ice 2 min') + + # Step 3: add 250 uL SOC medium + p300.transfer( + 250, + soc_reservoir['A1'], + cells_tube, + new_tip='always', + ) + + # Step 4: outgrowth (not simulated) + protocol.comment( + 'Incubate cells_rack 37°C 60 min at 250 rpm for outgrowth recovery') diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/reward.json new file mode 100644 index 0000000..b650ada --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_dna_addition": 1.0, "rubric_heat_shock": 1.0, "rubric_soc_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/trial.json new file mode 100644 index 0000000..a6737f0 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/ecoli-heat-shock-transformation/trial.json @@ -0,0 +1,30 @@ +{ + "task": "ecoli-heat-shock-transformation", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_dna_addition": 1.0, + "rubric_heat_shock": 1.0, + "rubric_soc_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.063752, + "input_tokens": 125053, + "output_tokens": 2093, + "started_at": "2026-10-07T17:53:20.245332Z", + "finished_at": "2026-10-07T17:54:48.169294Z", + "cache_tokens": 113664, + "claude_code_cost_usd": 0.16610199999999997 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/grader.json new file mode 100644 index 0000000..5196795 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/grader.json @@ -0,0 +1,235 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 602 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": false, + "detail": "[(395, 'D2'), (396, 'B1'), (397, 'B1')]" + }, + { + "name": "end_state:water", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dntp", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_pol", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_mm", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:template_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:rcutsmart", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dpni", + "pass": true, + "detail": "" + }, + { + "name": "end_state:t4_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:gg_enzyme", + "pass": true, + "detail": "" + }, + { + "name": "end_state:assembly_plate", + "pass": false, + "detail": "expected {'A1': 15.0, 'B1': 15.0, 'C1': 15.0, 'D1': 15.0}, got {'A1': 10.0, 'B1': 10.0, 'C1': 10.0, 'D1': 10.0}" + }, + { + "name": "end_state:cells_plate", + "pass": false, + "detail": "expected {'A1': 305.0, 'B1': 305.0, 'C1': 305.0, 'D1': 305.0}, got {'A1': 310.0, 'B1': 310.0, 'C1': 310.0, 'D1': 310.0}" + }, + { + "name": "end_state:lb_dextrose", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "Valid apiLevel 2.13 (via requirements), every volume uses a suitable pipette (p20 for \u226419 \u00b5L, p300 for 40/106/250 \u00b5L), no well exceeds capacity, and the simulator passed with no overdispense or empty-aspirate errors.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 0, + "evidence": "The no_cross_contamination check failed [(395,'D2'),(396,'B1'),(397,'B1')]: the gg_enzyme transfer uses new_tip='once' with mix_after=(5,15), so one tip mixes in assembly wells and then returns to the enzyme stock D2 and the next wells.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "The protocol deviates from the paper: fragments are eluted in 15 \u00b5L instead of 10 \u00b5L, the full 10 \u00b5L assembly eluate is transformed (the paper uses 5 \u00b5L), and the enzyme tip is reused across assembly wells; the end_state checks for assembly_plate and cells_plate also differ.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "pcr_setup", + "score": 1, + "evidence": "Master mix gives 19 \u00b5L per reaction, then 2.5 \u00b5L of each 1 \u00b5M primer and 1 \u00b5L of 0.5 ng/\u00b5L template give 25 \u00b5L, with the primer and template mapping taken from the design table.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dpni_and_cleanup", + "score": 1, + "evidence": "Comment records 98 C 30 s, 36 cycles with gradient annealing, 72 C 5 min; DpnI uses 19 \u00b5L water, 5 \u00b5L rCutSmart and 1 \u00b5L DpnI, with 37 C 30 min then 65 C 20 min; a pause for Zymo cleanup follows, in that order.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "assembly_mix", + "score": 1, + "evidence": "Each assembly gets 7 \u00b5L water, 2 \u00b5L 10X T4 buffer, 1 \u00b5L enzyme mix and fragments 3+2+3+2 \u00b5L from the design table, totalling 20 \u00b5L in its own well.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "cycling_and_transformation", + "score": 1, + "evidence": "Records a Golden Gate program (30 cycles of 37/16 C, 60 C, 4 C hold), cleanup eluted in 10 \u00b5L, transfer to TOP10 cells, 30 min ice, 42 C 60 s, 250 \u00b5L LB+dextrose, 37 C 60 min, plating on kanamycin.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 0.0, + "fidelity_to_paper": 0.0, + "pcr_setup": 1.0, + "dpni_and_cleanup": 1.0, + "assembly_mix": 1.0, + "cycling_and_transformation": 1.0 + }, + "summary": "The protocol is sound on volumes, PCR setup, DpnI, assembly mix and the cycling/transformation steps. It fails tips because the enzyme-mix transfer reuses one tip across mixing in assembly wells and the enzyme stock, which the measured contamination check flagged, and it fails fidelity because of the 15 \u00b5L fragment elution and the full 10 \u00b5L transformed.", + "level": "hard", + "score": 0.5, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "pcr_setup": 0.0625, + "dpni_and_cleanup": 0.0625, + "assembly_mix": 0.0625, + "cycling_and_transformation": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [ + "no_cross_contamination" + ], + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.8571, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.2167, + "judge_score": 0.5, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/protocol.py new file mode 100644 index 0000000..0a148fb --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/protocol.py @@ -0,0 +1,329 @@ +""" +Golden Gate assembly of four four-fragment chromoprotein expression plasmids +with the AssemblyTron workflow on the OT-2. + +Based on: Bradley et al., "AssemblyTron: flexible automation of DNA assembly +with Opentrons OT-2 lab robots", Synth. Biol. 2022, doi:10.1093/synbio/ysac032 +(Golden Gate script "based on Engler et al. (8)"; methods section 2.3). + +Workflow: + 1. Build a Q5 PCR master mix (buffer, dNTPs, polymerase, water) in + tubes_1_5ml_1 D1 and dispense it into pcr_plate A1:G1. + 2. Add primers (0.1 uM final, per paper) and linearized template (0.5 ng, + per paper) to give 25 uL PCRs for fragments 1-7. + 3. [manual] Gradient PCR in an off-deck thermocycler (the OT-2 thermocycler + has no gradient capability - paper section 3.1). + 4. DpnI digestion of the PCRs (19 uL water + 5 uL rCutSmart + 1 uL DpnI, + 30 min 37 C, 20 min 65 C - paper section 2.3). + 5. [manual] Zymo DNA Clean & Concentrator-5 cleanup of each fragment; + eluates returned to their pcr_plate wells. + 6. Set up four 20 uL Golden Gate reactions in assembly_plate A1:D1 with + fragment volumes proportional to length (fixed j5/AssemblyTron design). + 7. [manual] Golden Gate thermocycling (Engler-style program). + 8. [manual] Zymo cleanup of assemblies, eluted in 10 uL water (paper). + 9. Transform eluates into 50 uL TOP10 competent cells (cells_plate A1:D1), + 30 min on ice, 60 s heat shock at 42 C, +250 uL LB + 0.2% dextrose, + recover 60 min at 37 C, plate on LB + kanamycin (paper section 2.3). + +Volume choices derived from the paper: + * 25 uL PCR (paper 2.3): 5 uL 5X Q5 buffer (1X), 0.5 uL 10 mM dNTPs + (0.2 mM, Q5 standard), 2.5 uL each 1 uM primer (0.1 uM final, paper), + 1 uL template at 0.5 ng/uL (0.5 ng, paper), 0.25 uL Q5 polymerase + (0.5 U, NEB standard for 25 uL), water to 25 uL (13.25 uL). + Master mix = 19 uL/reaction, made for 8 reactions (7 fragments + 1 + spare, as stated on the deck) = water 106 uL, buffer 40 uL, dNTPs 4 uL, + polymerase 2 uL (152 uL total in D1). + * DpnI step exactly as paper 2.3: +19 uL water, +5 uL rCutSmart, +1 uL + DpnI per 25 uL PCR -> 50 uL. + * 20 uL Golden Gate reaction (fixed design gives 3+2+3+2 = 10 uL of + fragments). The paper does not tabulate the remaining 10 uL, so a sound + standard choice is made here: 2 uL 10X T4 DNA Ligase Buffer (1X final), + 1 uL Golden Gate Enzyme Mix (BsaI-HFv2 + T4 ligase), 7 uL water. +""" + +metadata = { + 'protocolName': 'AssemblyTron Golden Gate - 4x chromoprotein plasmids', + 'author': 'OT-2 protocol', + 'description': 'PCR of 7 fragments, DpnI, cleanup, 4-part Golden Gate ' + 'assembly x4, transformation into TOP10 (per AssemblyTron ' + 'paper, doi:10.1093/synbio/ysac032).', +} + +requirements = {'apiLevel': '2.13'} + + +def run(protocol): + protocol.set_rail_lights(True) + + # ---- labware (fixed deck) ---- + tubes_50ml_1 = protocol.load_labware( + 'opentrons_6_tuberack_falcon_50ml_conical', 1, label='tubes_50ml_1') + tubes_1_5ml_1 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 3, label='primer_plate') + template_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='template_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 5, label='pcr_plate') + assembly_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='assembly_plate') + cells_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 7, label='cells_plate') + tubes_15ml_1 = protocol.load_labware( + 'opentrons_15_tuberack_falcon_15ml_conical', 8, label='tubes_15ml_1') + + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + # ---- reagent locations ---- + water = tubes_50ml_1['A1'] + q5_buffer = tubes_1_5ml_1['A1'] # 5X Q5 reaction buffer + dntp = tubes_1_5ml_1['B1'] # 10 mM dNTPs + q5_pol = tubes_1_5ml_1['C1'] # Q5 High-Fidelity DNA Polymerase + pcr_mm = tubes_1_5ml_1['D1'] # empty at start -> master mix here + rcutsmart = tubes_1_5ml_1['A2'] + dpni = tubes_1_5ml_1['B2'] + t4_buffer = tubes_1_5ml_1['C2'] # 10X T4 DNA Ligase Buffer + gg_enzyme = tubes_1_5ml_1['D2'] # Golden Gate Enzyme Mix + lb_dextrose = tubes_15ml_1['A1'] # LB + 0.2% (w/v) dextrose + + # ---- fixed j5/AssemblyTron design ---- + # fragment number: (pcr well, fwd primer, rev primer, template well) + fragments = { + 1: ('A1', 'A1', 'A2', 'A1'), # tsPurple chromoprotein + 2: ('B1', 'B1', 'B2', 'A1'), # backbone + 3: ('C1', 'C1', 'C2', 'B1'), # YukonOFP chromoprotein + 4: ('D1', 'D1', 'D2', 'C1'), # aeBlue chromoprotein + 5: ('E1', 'E1', 'E2', 'A1'), # backbone / KanR + 6: ('F1', 'F1', 'F2', 'A1'), # backbone + 7: ('G1', 'G1', 'G2', 'D1'), # fuGFP chromoprotein + } + pcr_wells = [pcr_plate[fragments[f][0]] for f in range(1, 8)] + + # assemblies: well -> [(fragment number, volume uL), ...], 20 uL total + # (fragment volumes from the fixed design; reagents make up the rest: + # 7 uL water + 2 uL 10X T4 buffer + 1 uL GG enzyme mix) + assemblies = { + 'A1': [(2, 3.0), (5, 2.0), (6, 3.0), (1, 2.0)], # tsPurple + 'B1': [(2, 3.0), (5, 2.0), (6, 3.0), (3, 2.0)], # YukonOFP + 'C1': [(2, 3.0), (5, 2.0), (6, 3.0), (4, 2.0)], # aeBlue + 'D1': [(2, 3.0), (5, 2.0), (6, 3.0), (7, 2.0)], # fuGFP + } + + # ================================================================== + # STEP 1 - PCR master mix in tubes_1_5ml_1 D1 (for 8 reactions) + # 19 uL/reaction: 13.25 uL water + 5 uL 5X Q5 buffer + 0.5 uL 10 mM + # dNTPs + 0.25 uL Q5 polymerase. x8 = 106 + 40 + 4 + 2 = 152 uL. + # ================================================================== + protocol.comment('--- STEP 1: build Q5 PCR master mix (8 reactions) ' + 'in tubes_1_5ml_1 D1 ---') + p20.reset_tipracks() + p300.reset_tipracks() + p300.transfer(106, water, pcr_mm, new_tip='once') # water + p300.transfer(40, q5_buffer, pcr_mm, new_tip='once') # 5X buffer + p20.transfer(4, dntp, pcr_mm, new_tip='once') # dNTPs + p20.transfer(2, q5_pol, pcr_mm, new_tip='once') # Q5 pol + p300.pick_up_tip() + p300.mix(5, 100, pcr_mm) + p300.drop_tip() + + # ================================================================== + # STEP 2 - dispense master mix, primers and template -> 25 uL PCRs + # ================================================================== + protocol.comment('--- STEP 2: set up 7 x 25 uL PCRs in pcr_plate ' + 'A1:G1 ---') + p20.reset_tipracks() + # 19 uL master mix per reaction (p20; one tip, source is a common mix) + p20.transfer(19, pcr_mm, pcr_wells, new_tip='once') + + # 2.5 uL of each 1 uM primer -> 0.1 uM final (paper 2.3); + # fresh tip per primer to avoid cross-contamination + for f in range(1, 8): + pcr_w, fwd, rev, _ = fragments[f] + p20.transfer(2.5, primer_plate[fwd], pcr_plate[pcr_w], + new_tip='always') + p20.transfer(2.5, primer_plate[rev], pcr_plate[pcr_w], + new_tip='always') + + # 1 uL of 0.5 ng/uL linearized template -> 0.5 ng (paper 2.3) + for f in range(1, 8): + pcr_w, _, _, tmpl = fragments[f] + p20.transfer(1, template_plate[tmpl], pcr_plate[pcr_w], + new_tip='always') + + # mix each 25 uL reaction + p20.reset_tipracks() + for w in pcr_wells: + p20.pick_up_tip() + p20.mix(3, 15, w) + p20.drop_tip() + + # ================================================================== + # STEP 3 - [manual] gradient PCR off-deck (paper 2.3 & 3.1: the OT-2 + # thermocycler cannot do the annealing gradient, so tubes are moved to + # a Bio-Rad C100 gradient thermocycler; pcr_plate column 1 stands in + # for the 100 uL PCR tubes) + # ================================================================== + protocol.comment( + 'MANUAL - gradient PCR: seal pcr_plate / move the PCR tubes to a ' + 'gradient thermocycler (e.g. Bio-Rad C100) positioned per the ' + 'AssemblyTron instructions file. Run: 98 C 30 s; then 36 cycles of ' + '[98 C 10 s, 30 s at the AssemblyTron/j5 gradient annealing ' + 'temperature for each fragment, 72 C extension for the ' + 'AssemblyTron-calculated time (~20-30 s/kb)]; final extension 72 C ' + '5 min; hold 4 C. (Paper allows 34 or 36 cycles; 36 chosen as the ' + 'more robust option for 4-part assembly fragments.) Optionally take ' + 'a sample of each reaction for gel electrophoresis. Return the ' + 'reactions to the OT-2 deck (pcr_plate A1:G1) and resume.') + protocol.pause('Off-deck gradient PCR - see comment. Resume when the ' + 'PCR products are back in pcr_plate A1:G1.') + + # ================================================================== + # STEP 4 - DpnI digestion to destroy residual methylated template + # (paper 2.3: +19 uL water, +5 uL rCutSmart Buffer, +1 uL DpnI; + # 30 min 37 C, then 65 C 20 min inactivation) + # ================================================================== + protocol.comment('--- STEP 4: DpnI digestion of the 7 PCRs ---') + p20.reset_tipracks() + p20.transfer(19, water, pcr_wells, new_tip='once') + p20.transfer(5, rcutsmart, pcr_wells, new_tip='once') + p20.transfer(1, dpni, pcr_wells, new_tip='once') + # mix each 50 uL digestion + for w in pcr_wells: + p20.pick_up_tip() + p20.mix(5, 20, w) + p20.drop_tip() + + protocol.comment( + 'MANUAL - DpnI incubation: seal pcr_plate and incubate 30 min at ' + '37 C, then inactivate DpnI 20 min at 65 C (thermocycler block or ' + 'heat block; the robot cannot thermocycle this plate). Each well ' + 'now holds 50 uL.') + protocol.pause('DpnI: 30 min 37 C + 20 min 65 C off-deck. Resume when ' + 'done and the plate is back at ~room temperature.') + + # ================================================================== + # STEP 5 - [manual] Zymo column cleanup of each fragment + # Paper 2.3: DNA Clean and Concentrator-5, water elution. The paper + # elutes in 10 uL, but fragments 2 and 6 each donate 3 uL to four + # assemblies (12 uL total), so 15 uL elutions are specified here - + # a sound deviation guaranteeing enough cleaned fragment plus dead + # volume. + # ================================================================== + protocol.comment( + 'MANUAL - fragment cleanup: clean and concentrate each of the 7 ' + 'DpnI-treated fragments (50 uL each) with a Zymo DNA Clean & ' + 'Concentrator-5 column per the manufacturer protocol (binding ' + 'buffer, 2 washes) to remove polymerase, which interferes with ' + 'Golden Gate (paper 3.3). Elute each fragment in 15 uL ' + 'nuclease-free water (15 uL rather than the paper\'s 10 uL because ' + 'fragments 2 and 6 must each supply 4 x 3 uL = 12 uL to the four ' + 'assemblies). Return each eluate to its original well pcr_plate ' + 'A1:G1 and resume.') + protocol.pause('Zymo cleanup of fragments 1-7; eluates (15 uL each) ' + 'returned to pcr_plate A1:G1.') + + # ================================================================== + # STEP 6 - Golden Gate assembly setup (20 uL reactions, Engler-style) + # assembly_plate A1:D1. Reagents first, then fragments proportional to + # length (fixed design), enzyme mix last. + # ================================================================== + protocol.comment('--- STEP 6: set up 4 x 20 uL Golden Gate reactions ' + 'in assembly_plate A1:D1 ---') + p20.reset_tipracks() + asm_wells = [assembly_plate[w] for w in ['A1', 'B1', 'C1', 'D1']] + + # 7 uL water per reaction (choice documented in the module docstring) + p20.transfer(7, water, asm_wells, new_tip='once') + # 2 uL 10X T4 DNA Ligase Buffer -> 1X final + p20.transfer(2, t4_buffer, asm_wells, new_tip='once') + # fragments: shared fragments (2, 5, 6) distributed with one tip each + # (same source -> different destinations is contamination-safe); + # the unique chromoprotein fragment gets a fresh tip per assembly. + for frag, vol in [(2, 3.0), (5, 2.0), (6, 3.0)]: + src = pcr_plate[fragments[frag][0]] + p20.transfer(vol, src, asm_wells, new_tip='once') + for well_name, frags in assemblies.items(): + chromo_frag, chromo_vol = frags[3] + p20.transfer(chromo_vol, pcr_plate[fragments[chromo_frag][0]], + assembly_plate[well_name], new_tip='always') + # 1 uL Golden Gate Enzyme Mix (BsaI-HFv2 + T4 ligase) last, then mix + p20.transfer(1, gg_enzyme, asm_wells, new_tip='once', + mix_after=(5, 15)) + + # ================================================================== + # STEP 7 - [manual] Golden Gate thermocycling (paper 3.3: run in the + # Opentrons thermocycler module; the deck layout here has no module + # slot free, so this is recorded as a manual/off-deck step). Program + # based on Engler et al., as the AssemblyTron GG script is. + # ================================================================== + protocol.comment( + 'MANUAL - Golden Gate thermocycling: seal assembly_plate and run ' + 'in a thermocycler (Opentrons thermocycler module if available): ' + '30 cycles of [37 C 1 min, 16 C 1 min], then 60 C 5 min (BsaI/' + 'ligase heat inactivation), hold 4 C (program per Engler et al., ' + 'on which the AssemblyTron Golden Gate script is based; 30 cycles ' + 'suit a 4-fragment one-pot assembly). Return the plate to the deck ' + 'and resume.') + protocol.pause('Golden Gate thermocycling (30 x 37 C/16 C, 60 C 5 min, ' + 'hold 4 C). Resume when complete.') + + # ================================================================== + # STEP 8 - [manual] cleanup of the four assemblies (paper 2.3: + # Zymo DNA Clean and Concentrator-5, elute in 10 uL molecular grade + # water; eluates transformed into 50 uL TOP10) + # ================================================================== + protocol.comment( + 'MANUAL - assembly cleanup: clean and concentrate each 20 uL ' + 'Golden Gate reaction with a Zymo DNA Clean & Concentrator-5 ' + 'column and elute in 10 uL molecular grade water (paper 2.3). ' + 'Return each 10 uL eluate to its assembly_plate well A1:D1 and ' + 'resume. (The paper reserves 5 uL for NanoDrop quantification for ' + 'CFU/ug calculations; here the full 10 uL eluate is transformed - ' + 'quantify a separate aliquot if transformation efficiencies are ' + 'needed.)') + protocol.pause('Zymo cleanup of the 4 assemblies; 10 uL eluates ' + 'returned to assembly_plate A1:D1.') + + # ================================================================== + # STEP 9 - transformation into E. coli TOP10 (paper 2.3) + # ================================================================== + protocol.comment('--- STEP 9: transformation into TOP10 (cells_plate ' + 'A1:D1) ---') + p20.reset_tipracks() + p300.reset_tipracks() + cell_wells = [cells_plate[w] for w in ['A1', 'B1', 'C1', 'D1']] + for src, dst in zip(asm_wells, cell_wells): + # gentle mix - competent cells are shear-sensitive + p20.transfer(10, src, dst, new_tip='always', mix_after=(3, 10)) + + protocol.comment( + 'MANUAL - transformation incubations: keep cells_plate A1:D1 on ice ' + '30 min, heat shock 42 C for 60 s, then return to ice (~2 min). ' + '(Paper 2.3: 30 min on ice, 60 s at 42 C.)') + protocol.pause('Ice 30 min -> 42 C 60 s heat shock -> back on ice. ' + 'Resume with the plate back on the deck (slot 7).') + + # recovery medium: 250 uL LB + 0.2% dextrose per well (paper 2.3) + p300.reset_tipracks() + p300.transfer(250, lb_dextrose, cell_wells, new_tip='always', + mix_after=(3, 150)) + + protocol.comment( + 'MANUAL - recovery and plating: incubate cells_plate A1:D1 at 37 C ' + 'for 60 min with shaking (recovery in LB + 0.2% dextrose, paper ' + '2.3). Then plate 50-200 uL of each culture (neat or a 10x ' + 'dilution, depending on expected efficiency) onto LB agar + ' + 'kanamycin 50 ug/mL (fragment 5 carries KanR) and incubate at 37 C ' + 'overnight. Score colonies by chromoprotein colour (purple, ' + 'orange/Yukon, blue, green) and count CFU.') + + protocol.set_rail_lights(False) + protocol.comment('Protocol complete. Remove plates; final tip state: ' + 'no tips on either pipette.') diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/reward.json new file mode 100644 index 0000000..f45bc7b --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.8571, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.2167, "judge_score": 0.5, "critical_fail": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 0.0, "rubric_fidelity_to_paper": 0.0, "rubric_pcr_setup": 1.0, "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 1.0, "rubric_cycling_and_transformation": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/trial.json new file mode 100644 index 0000000..52a7bce --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly-hard/trial.json @@ -0,0 +1,31 @@ +{ + "task": "golden-gate-assembly-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.8571, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.2167, + "judge_score": 0.5, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + }, + "exception": null, + "cost_usd": 0.300086, + "input_tokens": 369155, + "output_tokens": 18728, + "started_at": "2026-10-07T17:42:48.552099Z", + "finished_at": "2026-10-07T17:52:15.296183Z", + "cache_tokens": 314624, + "claude_code_cost_usd": 0.898167 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/grader.json new file mode 100644 index 0000000..b901d43 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/grader.json @@ -0,0 +1,235 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 491 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": false, + "detail": "[(379, 'D2'), (380, 'B1'), (381, 'B1')]" + }, + { + "name": "end_state:water", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dntp", + "pass": true, + "detail": "" + }, + { + "name": "end_state:q5_pol", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_mm", + "pass": true, + "detail": "" + }, + { + "name": "end_state:primer_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:template_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:pcr_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:rcutsmart", + "pass": true, + "detail": "" + }, + { + "name": "end_state:dpni", + "pass": true, + "detail": "" + }, + { + "name": "end_state:t4_buffer", + "pass": true, + "detail": "" + }, + { + "name": "end_state:gg_enzyme", + "pass": true, + "detail": "" + }, + { + "name": "end_state:assembly_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:cells_plate", + "pass": true, + "detail": "" + }, + { + "name": "end_state:lb_dextrose", + "pass": true, + "detail": "" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "Valid apiLevel 2.13, pipettes match volume ranges, labware loaded with correct labels, and the end-state checks pass.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 0, + "evidence": "Measured check no_cross_contamination failed at (379,'D2') etc. Step 24 uses new_tip='once' with mix_after=(5,15), so the tip that mixed in assembly wells goes back into the gg_enzyme stock D2.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "fidelity_to_task", + "score": 0, + "evidence": "Step 24 reuses one tip with mix_after, which contaminates the enzyme stock and so deviates from a clean implementation. The mix volume of 15 \u00b5L is also arbitrary for 20 \u00b5L reactions.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "pcr_setup", + "score": 1, + "evidence": "Master mix 106/40/4/2 \u00b5L, 19 \u00b5L per well, 2.5 \u00b5L primers and 1 \u00b5L template paired as A1,A1,B1,C1,A1,A1,D1 with fresh tips each.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "dpni_and_cleanup", + "score": 1, + "evidence": "Steps 12-15 add 19 \u00b5L water, 5 \u00b5L rCutSmart and 1 \u00b5L DpnI, then mix; the incubation and Zymo cleanup are recorded with protocol.comment in order.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "assembly_mix", + "score": 1, + "evidence": "Assembly volumes are 7 \u00b5L water, 2 \u00b5L buffer, 3/2/3 \u00b5L backbone fragments, 2 \u00b5L chromoprotein fragments and 1 \u00b5L enzyme, matching the task.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "cycling_and_transformation", + "score": 1, + "evidence": "Golden Gate program, cleanup, 5 \u00b5L transfer to cells, 250 \u00b5L LB, and the ice/heat shock/recovery steps are all recorded or performed in order.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 0.0, + "fidelity_to_task": 0.0, + "pcr_setup": 1.0, + "dpni_and_cleanup": 1.0, + "assembly_mix": 1.0, + "cycling_and_transformation": 1.0 + }, + "summary": "The protocol simulates and gets nearly all quantities and orders right. However, step 24 uses one tip with mix_after across all assembly wells, which contaminates the enzyme stock (the measured contamination check failed), so the tips and fidelity items fail.", + "level": "easy", + "score": 0.5, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "pcr_setup": 0.0625, + "dpni_and_cleanup": 0.0625, + "assembly_mix": 0.0625, + "cycling_and_transformation": 0.0625 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [ + "no_cross_contamination" + ], + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9524, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.25, + "judge_score": 0.5, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_task": 0.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/protocol.py new file mode 100644 index 0000000..d367382 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/protocol.py @@ -0,0 +1,222 @@ +# Golden Gate assembly of four four-fragment chromoprotein expression plasmids +# with AssemblyTron on the OT-2 (Synth. Biol. 2022, doi:10.1093/synbio/ysac032). +# +# Steps 1-33 of the task: PCR master mix, primer/template addition, gradient PCR +# (manual comment), DpnI digestion, cleanup (manual comment), Golden Gate +# assembly, thermocycling (manual comment), cleanup (manual comment), and +# transformation of E. coli TOP10. + +metadata = {'apiLevel': '2.13'} + + +def run(protocol): + # Tip racks + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + # Pipettes + p20 = protocol.load_instrument( + 'p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument( + 'p300_single_gen2', 'right', tip_racks=[tips300]) + + # Labware + tubes_50ml_1 = protocol.load_labware( + 'opentrons_6_tuberack_falcon_50ml_conical', 1, label='tubes_50ml_1') + tubes_1_5ml_1 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap', 2, + label='tubes_1_5ml_1') + primer_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 3, label='primer_plate') + template_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 4, label='template_plate') + pcr_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 5, label='pcr_plate') + assembly_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 6, label='assembly_plate') + cells_plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 7, label='cells_plate') + tubes_15ml_1 = protocol.load_labware( + 'opentrons_15_tuberack_falcon_15ml_conical', 8, label='tubes_15ml_1') + + # Reagent sources + water = tubes_50ml_1['A1'] + q5_buffer = tubes_1_5ml_1['A1'] + dntp = tubes_1_5ml_1['B1'] + q5_pol = tubes_1_5ml_1['C1'] + pcr_mm = tubes_1_5ml_1['D1'] + rcutsmart = tubes_1_5ml_1['A2'] + dpni = tubes_1_5ml_1['B2'] + t4_buffer = tubes_1_5ml_1['C2'] + gg_enzyme = tubes_1_5ml_1['D2'] + lb_dextrose = tubes_15ml_1['A1'] + + # PCR reactions for fragments 1-7 in pcr_plate column 1 + pcr_wells = [pcr_plate[w] for w in + ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1']] + fwd_primers = [primer_plate[w] for w in + ['A1', 'B1', 'C1', 'D1', 'E1', 'F1', 'G1']] + rev_primers = [primer_plate[w] for w in + ['A2', 'B2', 'C2', 'D2', 'E2', 'F2', 'G2']] + templates = [template_plate[w] for w in + ['A1', 'A1', 'B1', 'C1', 'A1', 'A1', 'D1']] + + # Golden Gate assemblies A1:D1 and transformation destinations + asm_wells = [assembly_plate[w] for w in ['A1', 'B1', 'C1', 'D1']] + cells_wells = [cells_plate[w] for w in ['A1', 'B1', 'C1', 'D1']] + + # --- PCR master mix (8 reactions) --- + # Step 1: 106 uL water -> pcr_mm tube + p300.reset_tipracks() + p300.transfer(106, water, pcr_mm, new_tip='once') + # Step 2: 40 uL 5X Q5 reaction buffer -> pcr_mm tube + p300.reset_tipracks() + p300.transfer(40, q5_buffer, pcr_mm, new_tip='once') + # Step 3: 4 uL 10 mM dNTPs -> pcr_mm tube + p20.reset_tipracks() + p20.transfer(4, dntp, pcr_mm, new_tip='once') + # Step 4: 2 uL Q5 polymerase -> pcr_mm tube + p20.reset_tipracks() + p20.transfer(2, q5_pol, pcr_mm, new_tip='once') + # Step 5: mix master mix, 5 cycles at 100 uL + p300.reset_tipracks() + p300.pick_up_tip() + p300.mix(5, 100, pcr_mm) + p300.drop_tip() + + # --- PCR setup --- + # Step 6: 19 uL master mix to each PCR well + p20.reset_tipracks() + p20.transfer(19, pcr_mm, pcr_wells, new_tip='always') + # Step 7: 2.5 uL forward primer to each PCR well + p20.reset_tipracks() + p20.transfer(2.5, fwd_primers, pcr_wells, new_tip='always') + # Step 8: 2.5 uL reverse primer to each PCR well + p20.reset_tipracks() + p20.transfer(2.5, rev_primers, pcr_wells, new_tip='always') + # Step 9: 1 uL linearized template (0.5 ng/uL) to each PCR well + p20.reset_tipracks() + p20.transfer(1, templates, pcr_wells, new_tip='always') + # Step 10: mix each PCR, 3 cycles at 15 uL + p20.reset_tipracks() + for w in pcr_wells: + p20.pick_up_tip() + p20.mix(3, 15, w) + p20.drop_tip() + + # Step 11: gradient PCR (manual) + protocol.comment( + 'Seal pcr_plate (or move the PCR tubes) and transfer manually to the ' + 'Bio-Rad C100 gradient thermocycler. Run: 98C 30 s; 34 cycles of ' + '98C 10 s, annealing 30 s at the AssemblyTron/j5 optimal-gradient ' + 'temperature for each fragment, 72C extension at the time set by ' + 'AssemblyTron (about 20-30 s/kb); final extension 72C 5 min; hold ' + '4C. Take a sample of each reaction for gel electrophoresis, then ' + 'return to the OT-2.') + + # --- DpnI digestion of residual template --- + # Step 12: 19 uL water to each PCR well + p20.reset_tipracks() + p20.transfer(19, water, pcr_wells, new_tip='once') + # Step 13: 5 uL rCutSmart Buffer to each PCR well + p20.reset_tipracks() + p20.transfer(5, rcutsmart, pcr_wells, new_tip='once') + # Step 14: 1 uL DpnI to each PCR well + p20.reset_tipracks() + p20.transfer(1, dpni, pcr_wells, new_tip='once') + # Step 15: mix each PCR well, 3 cycles at 30 uL + p300.reset_tipracks() + for w in pcr_wells: + p300.pick_up_tip() + p300.mix(3, 30, w) + p300.drop_tip() + + # Step 16: DpnI incubation (manual) + protocol.comment( + 'Incubate pcr_plate A1:G1 at 37C for 30 min, then 65C for 20 min ' + '(DpnI inactivation) on the thermocycler block.') + + # Step 17: fragment cleanup (manual) + protocol.comment( + 'Pause the protocol. Clean and concentrate each of the 7 fragments ' + 'with a Zymo DNA Clean & Concentrator-5 column: 5:1 DNA Binding ' + 'Buffer to sample (250 uL per 50 uL), spin 30 s, wash 2 x 200 uL DNA ' + 'Wash Buffer (30 s spins), elute in 20 uL water after 1 min at room ' + 'temperature (30 s spin). Return the eluted fragments to their ' + 'original positions pcr_plate A1:G1 (fragments 1-7) and resume the ' + 'protocol.') + + # --- Golden Gate assembly (20 uL reactions, assembly_plate A1:D1) --- + # Step 18: 7 uL water to each assembly well + p20.reset_tipracks() + p20.transfer(7, water, asm_wells, new_tip='once') + # Step 19: 2 uL 10X T4 DNA Ligase Buffer to each assembly well + p20.reset_tipracks() + p20.transfer(2, t4_buffer, asm_wells, new_tip='once') + # Step 20: 3 uL fragment 2 (backbone) to each assembly well + p20.reset_tipracks() + p20.transfer(3, pcr_plate['B1'], asm_wells, new_tip='once') + # Step 21: 2 uL fragment 5 (backbone/KanR) to each assembly well + p20.reset_tipracks() + p20.transfer(2, pcr_plate['E1'], asm_wells, new_tip='once') + # Step 22: 3 uL fragment 6 (backbone) to each assembly well + p20.reset_tipracks() + p20.transfer(3, pcr_plate['F1'], asm_wells, new_tip='once') + # Step 23: 2 uL chromoprotein fragments 1, 3, 4, 7 (A1, C1, D1, G1) to + # assembly wells A1, B1, C1, D1 respectively + p20.reset_tipracks() + p20.transfer(2, [pcr_plate['A1'], pcr_plate['C1'], + pcr_plate['D1'], pcr_plate['G1']], + asm_wells, new_tip='always') + # Step 24: 1 uL Golden Gate Enzyme Mix to each assembly well, mix after + p20.reset_tipracks() + p20.transfer(1, gg_enzyme, asm_wells, new_tip='once', mix_after=(5, 15)) + + # Step 25: Golden Gate thermocycling (manual) + protocol.comment( + 'Run Golden Gate program on assembly_plate in the Opentrons ' + 'thermocycler module: 30 cycles of 37C 5 min then 16C 5 min; then ' + '60C 5 min; hold at 4C. Lid about 85C. If no module is available, ' + 'pause and move reactions to another thermocycler.') + + # Step 26: assembly cleanup (manual) + protocol.comment( + 'Clean and concentrate each assembly with a Zymo DNA Clean & ' + 'Concentrator-5 column (5:1 binding buffer, 2 x 200 uL wash) and ' + 'elute in 10 uL molecular grade water; return the eluates to ' + 'assembly_plate A1:D1.') + + # --- Transformation into E. coli TOP10 --- + # Step 27: 5 uL purified assembly to each cells well + p20.reset_tipracks() + p20.transfer(5, asm_wells, cells_wells, new_tip='always') + + # Step 28: NanoDrop quantification (manual) + protocol.comment( + 'Measure DNA concentration of the remaining 5 uL of each eluate with ' + 'a NanoDrop-2000c for CFU/ug calculation.') + + # Step 29: heat shock (manual) + protocol.comment( + 'Incubate cells_plate A1:D1 for 30 min on ice, heat shock at 42C for ' + '60 s, then return to ice for about 2 min.') + + # Step 30: 250 uL LB + 0.2% dextrose to each cells well + p300.reset_tipracks() + p300.transfer(250, lb_dextrose, cells_wells, new_tip='once') + + # Step 31: recovery (manual) + protocol.comment( + 'Recover cells at 37C for 60 min with shaking (about 250 rpm).') + + # Step 32: plating (manual) + protocol.comment( + 'Plate 50-200 uL of each recovery (neat or a 10x dilution, depending ' + 'on predicted efficiency) onto separate LB agar plates containing ' + 'kanamycin 50 ug/mL; incubate at 37C overnight.') + + # Step 33: colony scoring (manual) + protocol.comment( + 'Manually count colonies per plate and score the fraction showing ' + 'the expected chromoprotein colour (purple, orange, blue, green); ' + 'report CFU/ug of DNA plated.') diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/reward.json new file mode 100644 index 0000000..9e20ed7 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.9524, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 0.25, "judge_score": 0.5, "critical_fail": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 0.0, "rubric_fidelity_to_task": 0.0, "rubric_pcr_setup": 1.0, "rubric_dpni_and_cleanup": 1.0, "rubric_assembly_mix": 1.0, "rubric_cycling_and_transformation": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/trial.json new file mode 100644 index 0000000..c7dc095 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/golden-gate-assembly/trial.json @@ -0,0 +1,31 @@ +{ + "task": "golden-gate-assembly", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9524, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 0.25, + "judge_score": 0.5, + "critical_fail": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 0.0, + "rubric_fidelity_to_task": 0.0, + "rubric_pcr_setup": 1.0, + "rubric_dpni_and_cleanup": 1.0, + "rubric_assembly_mix": 1.0, + "rubric_cycling_and_transformation": 1.0 + }, + "exception": null, + "cost_usd": 0.167438, + "input_tokens": 212323, + "output_tokens": 7308, + "started_at": "2026-10-07T17:42:48.596562Z", + "finished_at": "2026-10-07T17:47:14.936895Z", + "cache_tokens": 172032, + "claude_code_cost_usd": 0.470171 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/grader.json new file mode 100644 index 0000000..4ec60c9 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/grader.json @@ -0,0 +1,234 @@ +{ + "variant": "reference", + "judge_model": "claude-sonnet-5-5", + "suspicious_tokens": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1333 + }, + "checks": { + "checks": [ + { + "name": "48_samples_to_odd_columns", + "pass": true, + "detail": "48 tubes -> 48 wells, odd columns only: True" + }, + { + "name": "step_order", + "pass": true, + "detail": "expected per well: beads > isopropanol > sample > remove > ethanol > remove > ethanol > remove > elution > recover" + }, + { + "name": "binding_volumes_40_250_250", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "mix_5x_after_sample", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "incubation_5min_before_magnet", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_4min_before_first_removal", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "supernatant_removed_each_step", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_engaged_for_all_removals", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "two_500ul_ethanol_washes", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "air_dry_4min", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_100ul", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_off_magnet_then_90s_on", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "recover_70_100ul_one_well_each", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "distinct_elution_wells", + "pass": true, + "detail": "48 distinct elution wells" + }, + { + "name": "elution_plate_4C_before_recovery", + "pass": true, + "detail": "temperatures set: [(0.0, 4.0), (900.0, 4.0)]" + }, + { + "name": "fresh_tip_per_sample_no_cross_contact", + "pass": true, + "detail": "48 distinct tips for 48 sample transfers; violations: []" + }, + { + "name": "reservoir_columns_within_15ml", + "pass": true, + "detail": "uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 12000, 10: 12000, 11: 12000, 12: 12000" + } + ], + "checks_passed": 17, + "checks_total": 17, + "total_delay_min": 15.0, + "example_well": "A1", + "example_sequence": [ + "beads 40uL", + "isopropanol 250uL", + "sample 250uL", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "elution 100uL", + "recover 100uL (magnet)" + ] + }, + "critical_failures": [], + "judge": { + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.13, all volumes within pipette ranges (multi chunks \u2264200 \u00b5L, p1000 250 \u00b5L), wells within capacity, reservoir draws within 15 mL per measured check.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "Check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips; every tip is dropped and a fresh tip is used per column or per sample.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_paper", + "score": 0, + "evidence": "No mixing after elution buffer addition, and only 500 \u00b5L of the 540 \u00b5L supernatant is removed, leaving about 40 \u00b5L. Both are deviations or additions, and this item is all-or-nothing.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "sample_handling", + "score": 1, + "evidence": "Check 48_samples_to_odd_columns passed. sample_routes pairs each tube with its own well in columns 1,3,5,7,9,11, and each eluate goes to a distinct elution well.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding_and_separation", + "score": 1, + "evidence": "Beads 40, isopropanol 250, then sample 250 \u00b5L, mixed 5x, 5 min delay, magnet engaged with a 4 min delay, then removal to the waste plate with the magnet on. The measured checks all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "washes_and_drying", + "score": 1, + "evidence": "Two 500 \u00b5L ethanol washes, each added and removed with the magnet engaged. Then a 4 min delay before magdeck.disengage(). Checks two_500ul_ethanol_washes and air_dry_4min passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "elution_recovery", + "score": 0, + "evidence": "Elution buffer is dispensed with the magnet off but never mixed. There is only a 30 s delay, then the magnet is engaged, so the rubric's 'added off-magnet and mixed' is not met, although recovery of 100 \u00b5L to distinct wells is fine.", + "votes": [ + 0, + 0, + 0 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_paper": 0.0, + "sample_handling": 1.0, + "binding_and_separation": 1.0, + "washes_and_drying": 1.0, + "elution_recovery": 0.0 + }, + "summary": "The protocol passes all 17 measured checks. Sample placement, binding, washes and drying, tip handling and recovery to distinct elution wells are correct. It fails elution_recovery because the elution buffer is never mixed. It fails fidelity_to_paper because that mixing is missing and about 40 \u00b5L of supernatant is left behind at each removal.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13, all volumes within pipette ranges (multi chunks \\u2264200 \\u00b5L, p1000 250 \\u00b5L), wells within capacity, reservoir draws within 15 mL per measured check.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips; every tip is dropped and a fresh tip is used per column or per sample.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"No mixing after elution buffer addition, and only 500 \\u00b5L of the 540 \\u00b5L supernatant is removed, leaving about 40 \\u00b5L. Both are deviations or additions, and this item is all-or-nothing.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed. sample_routes pairs each tube with its own well in columns 1,3,5,7,9,11, and each eluate goes to a distinct elution well.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40, isopropanol 250, then sample 250 \\u00b5L, mixed 5x, 5 min delay, magnet engaged with a 4 min delay, then removal to the waste plate with the magnet on. The measured checks all passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes, each added and removed with the magnet engaged. Then a 4 min delay before magdeck.disengage(). Checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"Elution buffer is dispensed with the magnet off but never mixed. There is only a 30 s delay, then the magnet is engaged, so the rubric's 'added off-magnet and mixed' is not met, although recovery of 100 \\u00b5L to distinct wells is fine.\"}], \"summary\": \"The protocol passes all 17 measured checks. Sample placement, binding, washes and drying, tip handling and recovery to distinct elution wells are correct. It fails elution_recovery because the elution buffer is never mixed. It fails fidelity_to_paper because that mixing is missing and about 40 \\u00b5L of supernatant is left behind at each removal.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13; all volumes within pipette ranges (p1000 250 \\u00b5L, multi \\u2264200 \\u00b5L chunks, 40 \\u00b5L beads); 17/17 checks passed and reservoir columns stay within 15 mL.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips; every pick_up_tip is paired with drop_tip and per-column fresh tips are used for sample-contacting steps.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"The beads are never resuspended or mixed after the 100 \\u00b5L elution buffer is added (only a delay, then the magnet engages), which departs from the paper's 'beads are resuspended in 100 \\u00b5L of elution buffer'.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed. The p1000 moves each 250 \\u00b5L sample into its own well of columns 1,3,5,7,9,11 with a fresh tip and a distinct elution well per sample.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Checks binding_volumes_40_250_250, mix_5x_after_sample, incubation_5min, magnet_4min and magnet_engaged_for_all_removals all passed; the order is beads, isopropanol, sample.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Checks two_500ul_ethanol_washes and air_dry_4min passed; the removals are done with the magnet engaged, then magdeck.disengage() is called before elution.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"Elution buffer is dispensed with the magnet off but never mixed. The rubric requires it to be added off-magnet and mixed, and the code only waits 30 s before engaging.\"}], \"summary\": \"The protocol passed all 17 measured run-log checks and handles binding, separation, washes, drying and recovery of the eluate to distinct 4 \\u00b0C wells correctly. It fails the elution step and fidelity because the beads are never mixed or resuspended after the 100 \\u00b5L of elution buffer is added.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 1, \"evidence\": \"apiLevel 2.13, p1000 handles 250 \\u00b5L, p300 multi splits into \\u2264200 \\u00b5L aspirations, wells hold \\u2264540 \\u00b5L, reservoir columns within 15 mL (check passed), modules used correctly.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"Check fresh_tip_per_sample_no_cross_contact passed (48 distinct tips); every pickup is followed by drop_tip, and shared-reagent tips are reused only for reagent dispensing.\"}, {\"id\": \"fidelity_to_paper\", \"score\": 0, \"evidence\": \"The paper says the beads are resuspended in 100 \\u00b5L elution buffer, but the code only dispenses the buffer (dispense_to_column) with no mixing, so the beads are never resuspended.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed; sample_routes zips tubes to odd-column wells with a fresh p1000 tip each, and elution wells are distinct.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"Beads 40, isopropanol 250 and sample 250 added in that order, p1000.mix(5, 250), 5 min delay, engage with 4 min delay, then 500 \\u00b5L removed to waste; the checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two 500 \\u00b5L ethanol washes, each added and removed with the magnet engaged, then a 4 min delay with the magnet on and disengage before elution; checks passed.\"}, {\"id\": \"elution_recovery\", \"score\": 0, \"evidence\": \"The 100 \\u00b5L elution buffer is added off-magnet but never mixed (no mix call after dispense_to_column in step 7), though 90 s magnet and a 100 \\u00b5L recovery to distinct wells on the 4 \\u00b0C plate do pass.\"}], \"summary\": \"The protocol passes all 17 measured checks and handles tips, binding and washes correctly. It never mixes or resuspends the beads after adding the elution buffer, so elution_recovery and fidelity_to_paper fail.\"}]", + "level": "hard", + "score": 0.6875, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_paper": 0.25, + "sample_handling": 0.0625, + "binding_and_separation": 0.0625, + "washes_and_drying": 0.0625, + "elution_recovery": 0.0625 + }, + "model": "anthropic/claude-sonnet-5.5", + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "rewards": { + "reward": 0.6875, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 0.6875, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 0.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/protocol.py new file mode 100644 index 0000000..d621489 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/protocol.py @@ -0,0 +1,432 @@ +""" +Automated in-house magnetic-bead RNA extraction of 48 samples on an Opentrons OT-2. + +Reimplementation of the "OT-2 in-house" protocol of: + + Lazaro-Perona F, Rodriguez-Antolin C, Alguacil-Guillen M, Gutierrez-Arroyo A, + Mingorance J, Garcia-Rodriguez J, et al. "Evaluation of two automated low-cost + RNA extraction protocols for SARS-CoV-2 detection." PLoS ONE 2021;16(2):e0246302. + doi:10.1371/journal.pone.0246302 + +Recipe per sample ("OT-2 in-house protocol" steps 1-9 and Table 1 of the paper): + + 1. 40 uL magnetic beads + 250 uL isopropanol + 250 uL inactivated sample; + mix by pipetting five times; incubate 5 min at room temperature + 2. engage the GEN1 magnetic module for 4 min + 3. collect the supernatant and discard it + 4. wash 1: add 500 uL ethanol 70%, collect and discard + 5. wash 2: add 500 uL ethanol 70%, collect and discard + 6. air dry for 4 min (magnets engaged) + 7. disengage the magnets and add 100 uL elution buffer + 8. after 30 s engage the magnets again + 9. after 90 s collect the supernatant (eluted RNA) into the elution plate + +The binding mixture is 40 + 250 + 250 = 540 uL per well; at step 3, 500 uL of +supernatant is removed so that the bead pellet, held against the wall by the magnet, +is never aspirated (the paper: "magnetic beads are pulled to one side of the tubes +with a magnet and supernatant is discarded"). The washes at steps 4 and 5 add and +remove 500 uL each. + +Deck (fixed by the operator) +---------------------------- + 1 waste plate (supernatant + washes) usascientific_96_wellplate_2.4ml_deep + 2 200 uL filter tips opentrons_96_filtertiprack_200ul + 3 200 uL filter tips opentrons_96_filtertiprack_200ul + 4 Magnetic Module GEN1 + extraction plate (usascientific_96_wellplate_2.4ml_deep) + 5 reagent reservoir nest_12_reservoir_15ml + column 2 magnetic beads + column 4 elution buffer + columns 6-7 isopropanol + columns 9-12 ethanol 70% + 6 Temperature Module GEN1 + elution plate (thermo_96_wellplate_200ul), 4 C + 7 samples 25-48 (2 mL tubes) opentrons_24_tuberack_..._snapcap + 9 200 uL filter tips opentrons_96_filtertiprack_200ul + 10 samples 1-24 (2 mL tubes) opentrons_24_tuberack_..._snapcap + 11 1000 uL filter tips opentrons_96_filtertiprack_1000ul + 12 fixed trash + +Pipettes: p1000_single_gen2 (left), p300_multi_gen2 (right). + +The 48 samples occupy the odd columns (1, 3, 5, 7, 9, 11) of the extraction plate, +one sample per well, and each eluate is recovered into the matching odd column of +the elution plate. The 8-channel pipette is used for everything that happens in a +plate (the 200 uL filter tips cap it at 200 uL per aspiration, so larger volumes are +split); the single-channel p1000 is used for the tube -> plate sample transfer, with +a fresh tip per sample. + +Tip budget: 8-channel 256 of the 288 available 200 uL tips, single-channel 48 of the +96 available 1000 uL tips. +""" + +from opentrons import protocol_api +from opentrons.types import Point + +metadata = { + 'protocolName': 'SARS-CoV-2 RNA extraction - in-house magnetic beads - 48 samples', + 'author': 'Hospital Universitario La Paz, Madrid (reimplemented)', + 'source': 'PLoS ONE 2021;16(2):e0246302 - OT-2 in-house protocol', + 'apiLevel': '2.13', +} + + +# ---------------------------------------------------------------------------- +# Protocol constants (values as described in the paper) +# ---------------------------------------------------------------------------- + +SAMPLE_COUNT = 48 +CHANNELS = 8 +# Odd columns of the extraction / waste / elution plates hold the samples. +PLATE_COLUMNS = ['1', '3', '5', '7', '9', '11'] + +# Reagent volumes per sample, uL (paper Table 1, "OT-2 in-house" column). +BEADS_VOLUME = 40.0 # magnetic beads, Mag-Bind TotalPure NGS +ISOPROPANOL_VOLUME = 250.0 # isopropanol absolute +SAMPLE_VOLUME = 250.0 # inactivated sample +SUPERNATANT_VOLUME = 500.0 # removed after binding (540 uL mix, pellet spared) +WASH_VOLUME = 500.0 # ethanol 70%, per wash +WASH_COUNT = 2 # two washes +ELUTION_VOLUME = 100.0 # elution buffer, recovered in full + +# Mixing, "mix by pipetting five times". +MIX_REPETITIONS = 5 +MIX_VOLUME = 250.0 + +# Incubation / magnet / drying times. +BIND_INCUBATION_MIN = 5 # step 1: 5 min at room temperature +MAGNET_SEPARATION_MIN = 4 # step 2: magnetic module activated 4 min +AIR_DRY_MIN = 4 # step 6: air dry 4 min +ELUTION_INCUBATION_SEC = 30 # step 8: 30 s off the magnet +ELUTION_SEPARATION_SEC = 90 # step 9: 90 s on the magnet + +ELUTION_TEMPERATURE = 4.0 # temperature module set point, C + +# Liquid-handling parameters. +MULTI_TIP_VOLUME = 200.0 # 8-channel runs 200 uL filter tips +RESERVOIR_ASPIRATE_Z = 1.5 # mm above the reservoir bottom +PLATE_DISPENSE_Z = 5.0 # mm above the well bottom when dispensing reagents +SUPER_ASPIRATE_START_Z = 3.0 # first supernatant aspiration, mm above the bottom +SUPER_ASPIRATE_END_Z = 0.5 # last supernatant aspiration, mm above the bottom +BEAD_SIDE_OFFSET_X = 0.0 # mm: lateral offset for supernatant aspiration. + # On the GEN1 module the magnets sit at the + # interstices between wells and hold the pellet + # against the wall(s), so the tip aspirates from + # the centre of the well. Set to +/- a couple of + # mm if your module pellets the beads on one side + # only, to move the tip away from the pellet. +WASTE_DISPENSE_Z = -5.0 # top(z=-5) of the waste well +SLOW_RATE = 0.4 # flow-rate multiplier for supernatant removal +BLOWOUT_Z = -5.0 # top(z=-5) for blow outs + +# Nominal volumes the operator has loaded into each reagent column of the +# reservoir, uL. Isopropanol and ethanol are spread over several columns; the +# protocol empties one column before moving to the next. +BEADS_COLUMNS = ([2], [2200.0]) # 48 x 40 = 1920 uL needed +ELUTION_COLUMNS = ([4], [5000.0]) # 48 x 100 = 4800 uL needed +ISOPROPANOL_COLUMNS = ([6, 7], [6200.0, 6200.0]) # 48 x 250 = 12000 uL needed +ETHANOL_COLUMNS = ([9, 10, 11, 12], [12500.0] * 4) # 48 x 500 x 2 = 48000 uL needed + + +# ---------------------------------------------------------------------------- +# Helpers +# ---------------------------------------------------------------------------- + +def split_volume(total, max_per_aspiration): + """Split ``total`` uL into chunks that fit in one tip (8-channel: 200 uL).""" + chunks = [] + remaining = float(total) + while remaining > max_per_aspiration + 1e-6: + chunks.append(max_per_aspiration) + remaining -= max_per_aspiration + if remaining > 1e-6: + chunks.append(round(remaining, 6)) + return chunks + + +class ReagentSource: + """A reagent that occupies one or more columns of the reservoir. + + Tracks what has been aspirated from each column and always returns a column + that still holds the requested volume, so a column is never run dry. + """ + + def __init__(self, reservoir, columns, volumes, name): + self.name = name + self._wells = [reservoir.wells_by_name()['A{}'.format(c)] for c in columns] + self._remaining = [float(v) for v in volumes] + + def take(self, volume): + """Book ``volume`` uL and return the well it should be aspirated from.""" + for index, remaining in enumerate(self._remaining): + if remaining + 1e-6 >= volume: + self._remaining[index] = remaining - volume + return self._wells[index] + raise RuntimeError( + 'Not enough {} left in the reservoir ({} uL requested).'.format( + self.name, volume)) + + @property + def current_well(self): + """The first column that still holds reagent (for resuspending).""" + for well, remaining in zip(self._wells, self._remaining): + if remaining > 1e-6: + return well + raise RuntimeError('The {} reservoir is empty.'.format(self.name)) + + +def dispense_to_column(pip, volume_per_well, source, dest_well): + """Dispense ``volume_per_well`` uL of reagent into each well of a column. + + ``source`` is a ReagentSource; ``dest_well`` is the top (row A) well of the + destination column - the 8 tips cover the remaining seven wells. Volumes + larger than the 200 uL tip are split over several aspirations. + """ + for chunk in split_volume(volume_per_well, MULTI_TIP_VOLUME): + source_well = source.take(chunk * CHANNELS) + pip.aspirate(chunk, source_well.bottom(z=RESERVOIR_ASPIRATE_Z)) + pip.dispense(chunk, dest_well.bottom(z=PLATE_DISPENSE_Z)) + pip.blow_out(dest_well.top(z=BLOWOUT_Z)) + + +def remove_from_column(pip, volume_per_well, source_well, dest_well): + """Aspirate ``volume_per_well`` uL from each well of a column and discard it. + + Used with the magnets engaged: the bead pellet is held against the well wall, + so every aspiration is done slowly from the centre of the well (see + BEAD_SIDE_OFFSET_X) and the tip follows the falling liquid level, leaving the + pellet undisturbed. + """ + chunks = split_volume(volume_per_well, MULTI_TIP_VOLUME) + last = len(chunks) - 1 + for index, chunk in enumerate(chunks): + z = SUPER_ASPIRATE_START_Z + ( + (SUPER_ASPIRATE_END_Z - SUPER_ASPIRATE_START_Z) * index / last + if last else 0.0) + pip.aspirate( + chunk, + source_well.bottom(z=z).move(Point(x=BEAD_SIDE_OFFSET_X)), + rate=SLOW_RATE) + pip.dispense(chunk, dest_well.top(z=WASTE_DISPENSE_Z)) + pip.blow_out(dest_well.top(z=WASTE_DISPENSE_Z)) + + +def run(protocol: protocol_api.ProtocolContext): + # ------------------------------------------------------------------ deck + waste_plate = protocol.load_labware( + 'usascientific_96_wellplate_2.4ml_deep', '1', + label='Waste plate (supernatant and washes)') + tipracks_200 = [ + protocol.load_labware('opentrons_96_filtertiprack_200ul', '2'), + protocol.load_labware('opentrons_96_filtertiprack_200ul', '3'), + protocol.load_labware('opentrons_96_filtertiprack_200ul', '9'), + ] + magdeck = protocol.load_module('magnetic module', '4') + extraction_plate = magdeck.load_labware( + 'usascientific_96_wellplate_2.4ml_deep', label='Extraction plate') + reservoir = protocol.load_labware( + 'nest_12_reservoir_15ml', '5', label='Reagent reservoir') + tempdeck = protocol.load_module('tempdeck', '6') + elution_plate = tempdeck.load_labware( + 'thermo_96_wellplate_200ul', label='Elution plate') + sample_tubes_25_48 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '7', + label='Samples 25-48') + sample_tubes_1_24 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '10', + label='Samples 1-24') + tiprack_1000 = protocol.load_labware('opentrons_96_filtertiprack_1000ul', '11') + + p1000 = protocol.load_instrument( + 'p1000_single_gen2', 'left', tip_racks=[tiprack_1000]) + p300m = protocol.load_instrument( + 'p300_multi_gen2', 'right', tip_racks=tipracks_200) + + # --------------------------------------------------------- module set-up + # The elution plate is kept at 4 C for the whole run and the magnets start + # retracted. + magdeck.disengage() + tempdeck.set_temperature(ELUTION_TEMPERATURE) + protocol.comment('Cooling the elution plate on the Temperature Module to ' + '{:.0f} C.'.format(ELUTION_TEMPERATURE)) + tempdeck.await_temperature(ELUTION_TEMPERATURE) + + # --------------------------------------------------------- reagent map + beads = ReagentSource(reservoir, BEADS_COLUMNS[0], BEADS_COLUMNS[1], + 'magnetic beads') + elution_buffer = ReagentSource(reservoir, ELUTION_COLUMNS[0], ELUTION_COLUMNS[1], + 'elution buffer') + isopropanol = ReagentSource(reservoir, ISOPROPANOL_COLUMNS[0], + ISOPROPANOL_COLUMNS[1], 'isopropanol') + ethanol = ReagentSource(reservoir, ETHANOL_COLUMNS[0], ETHANOL_COLUMNS[1], + 'ethanol 70%') + + # Sample routing: extraction-plate column k takes the four tubes of column k + # of the slot-10 rack (samples 1-24) and the four tubes of column k of the + # slot-7 rack (samples 25-48), one sample per well, rows A-H. + sample_routes = [] + for index, column in enumerate(PLATE_COLUMNS): + tubes = sample_tubes_1_24.columns()[index] + sample_tubes_25_48.columns()[index] + wells = extraction_plate.columns_by_name()[column] + sample_routes += list(zip(tubes, wells)) + if len(sample_routes) != SAMPLE_COUNT: + raise RuntimeError('Expected {} sample routes, built {}.'.format( + SAMPLE_COUNT, len(sample_routes))) + + waste_columns = {c: waste_plate.columns_by_name()[c][0] for c in PLATE_COLUMNS} + elution_columns = {c: elution_plate.columns_by_name()[c][0] for c in PLATE_COLUMNS} + extraction_columns = {c: extraction_plate.columns_by_name()[c][0] + for c in PLATE_COLUMNS} + + # ================================================================== + # Step 1a - 40 uL of magnetic beads into every sample well. + # Beads and isopropanol are dispensed before the samples, so the tips + # only ever touch shared reagent: one tip set serves all six columns. + # ================================================================== + protocol.comment('Step 1a: {} uL of magnetic beads per well.'.format( + int(BEADS_VOLUME))) + p300m.pick_up_tip() + for column in PLATE_COLUMNS: + # Keep the bead suspension homogeneous before aspirating from it. + p300m.mix(3, 100.0, + beads.current_well.bottom(z=RESERVOIR_ASPIRATE_Z + 1.0)) + dispense_to_column(p300m, BEADS_VOLUME, beads, extraction_columns[column]) + p300m.drop_tip() + + # ================================================================== + # Step 1b - 250 uL of isopropanol into every sample well. + # ================================================================== + protocol.comment('Step 1b: {} uL of isopropanol per well.'.format( + int(ISOPROPANOL_VOLUME))) + p300m.pick_up_tip() + for column in PLATE_COLUMNS: + dispense_to_column(p300m, ISOPROPANOL_VOLUME, isopropanol, + extraction_columns[column]) + p300m.drop_tip() + + # ================================================================== + # Step 1c - 250 uL of each inactivated sample, then mix by pipetting + # five times. Fresh 1000 uL filter tip for every sample. + # ================================================================== + protocol.comment('Step 1c: {} uL of each of the {} samples, mixed {} times ' + 'with {} uL.'.format(int(SAMPLE_VOLUME), SAMPLE_COUNT, + MIX_REPETITIONS, int(MIX_VOLUME))) + p1000.flow_rate.aspirate = 150.0 + p1000.flow_rate.dispense = 150.0 + for number, (tube, well) in enumerate(sample_routes, start=1): + p1000.pick_up_tip() + p1000.aspirate(SAMPLE_VOLUME, tube.bottom(z=1.0)) + p1000.dispense(SAMPLE_VOLUME, well.bottom(z=PLATE_DISPENSE_Z)) + p1000.blow_out(well.top(z=BLOWOUT_Z)) + p1000.mix(MIX_REPETITIONS, MIX_VOLUME, well.bottom(z=2.0)) + p1000.blow_out(well.top(z=BLOWOUT_Z)) + p1000.touch_tip(well, v_offset=BLOWOUT_Z) + p1000.drop_tip() + if number % 8 == 0: + protocol.comment(' {}/{} samples loaded.'.format(number, SAMPLE_COUNT)) + + # ================================================================== + # Step 1d - incubate 5 min at room temperature. + # ================================================================== + protocol.comment('Step 1d: incubating {} min at room temperature.'.format( + BIND_INCUBATION_MIN)) + protocol.delay(minutes=BIND_INCUBATION_MIN, + msg='Binding incubation, {} min at room temperature'.format( + BIND_INCUBATION_MIN)) + + # ================================================================== + # Step 2 - engage the magnetic module for 4 min. + # ================================================================== + protocol.comment('Step 2: engaging the Magnetic Module for {} min.'.format( + MAGNET_SEPARATION_MIN)) + magdeck.engage() + protocol.delay(minutes=MAGNET_SEPARATION_MIN, + msg='Magnetic separation, {} min'.format(MAGNET_SEPARATION_MIN)) + + # ================================================================== + # Step 3 - collect the supernatant and discard it (500 uL per well). + # Fresh tips: the supernatant carries the sample. + # ================================================================== + protocol.comment('Step 3: removing {} uL of supernatant per well.'.format( + int(SUPERNATANT_VOLUME))) + for column in PLATE_COLUMNS: + p300m.pick_up_tip() + remove_from_column(p300m, SUPERNATANT_VOLUME, extraction_columns[column], + waste_columns[column]) + p300m.drop_tip() + + # ================================================================== + # Steps 4 and 5 - two washes with 500 uL of ethanol 70%: add, collect + # and discard. Each column gets a fresh tip set, which is used both to + # dispense the ethanol and to remove it (same wells, so no carry-over). + # ================================================================== + for wash in range(1, WASH_COUNT + 1): + protocol.comment('Step {}: ethanol 70% wash - {} uL in, {} uL out, per ' + 'well.'.format(wash + 3, int(WASH_VOLUME), + int(WASH_VOLUME))) + for column in PLATE_COLUMNS: + p300m.pick_up_tip() + dispense_to_column(p300m, WASH_VOLUME, ethanol, + extraction_columns[column]) + remove_from_column(p300m, WASH_VOLUME, extraction_columns[column], + waste_columns[column]) + p300m.drop_tip() + + # ================================================================== + # Step 6 - air dry for 4 min with the magnets engaged. + # ================================================================== + protocol.comment('Step 6: air drying the beads for {} min, magnets ' + 'engaged.'.format(AIR_DRY_MIN)) + protocol.delay(minutes=AIR_DRY_MIN, + msg='Air dry, {} min'.format(AIR_DRY_MIN)) + + # ================================================================== + # Step 7 - disengage the magnets and add 100 uL of elution buffer. + # Fresh tips per column: the tips touch the beads. + # ================================================================== + protocol.comment('Step 7: disengaging the magnets and adding {} uL of elution ' + 'buffer per well.'.format(int(ELUTION_VOLUME))) + magdeck.disengage() + for column in PLATE_COLUMNS: + p300m.pick_up_tip() + dispense_to_column(p300m, ELUTION_VOLUME, elution_buffer, + extraction_columns[column]) + p300m.drop_tip() + + # ================================================================== + # Step 8 - after 30 s, engage the magnets again. + # ================================================================== + protocol.comment('Step 8: {} s off the magnets, then engaging for {} s.'.format( + ELUTION_INCUBATION_SEC, ELUTION_SEPARATION_SEC)) + protocol.delay(seconds=ELUTION_INCUBATION_SEC, + msg='Elution incubation, {} s, magnets disengaged'.format( + ELUTION_INCUBATION_SEC)) + magdeck.engage() + + # ================================================================== + # Step 9 - after 90 s, collect the supernatant and transfer it to the + # elution plate on the Temperature Module (4 C), one well per sample. + # ================================================================== + protocol.delay(seconds=ELUTION_SEPARATION_SEC, + msg='Magnetic separation of the eluate, {} s'.format( + ELUTION_SEPARATION_SEC)) + protocol.comment('Step 9: recovering {} uL of eluate per sample into the ' + 'elution plate at {:.0f} C.'.format(int(ELUTION_VOLUME), + ELUTION_TEMPERATURE)) + tempdeck.set_temperature(ELUTION_TEMPERATURE) + for column in PLATE_COLUMNS: + p300m.pick_up_tip() + for chunk in split_volume(ELUTION_VOLUME, MULTI_TIP_VOLUME): + p300m.aspirate( + chunk, + extraction_columns[column].bottom(z=SUPER_ASPIRATE_END_Z).move( + Point(x=BEAD_SIDE_OFFSET_X)), + rate=SLOW_RATE) + p300m.dispense(chunk, elution_columns[column].bottom(z=2.0)) + p300m.blow_out(elution_columns[column].top(z=-2.0)) + p300m.drop_tip() + + # ------------------------------------------------------------------ end + magdeck.disengage() + protocol.comment('Run finished: {} eluates in the odd columns of the elution ' + 'plate on the Temperature Module (kept at {:.0f} C), magnets ' + 'disengaged. Remove the elution plate when ready.'.format( + SAMPLE_COUNT, ELUTION_TEMPERATURE)) diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/reward.json new file mode 100644 index 0000000..a1137c3 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/reward.json @@ -0,0 +1 @@ +{"reward": 0.6875, "sim_pass": 1.0, "checks_frac": 1.0, "judge_score": 0.6875, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 0.0, "deterministic_reward": 1.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_paper": 0.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 0.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/trial.json new file mode 100644 index 0000000..4dee377 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction-hard/trial.json @@ -0,0 +1,32 @@ +{ + "task": "opentrons-rna-extraction-hard", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.6875, + "sim_pass": 1.0, + "checks_frac": 1.0, + "judge_score": 0.6875, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 0.0, + "deterministic_reward": 1.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_paper": 0.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 0.0 + }, + "exception": null, + "cost_usd": 1.436312, + "input_tokens": 3467241, + "output_tokens": 59681, + "started_at": "2026-10-07T17:54:48.193353Z", + "finished_at": "2026-10-07T18:37:27.847812Z", + "cache_tokens": 3346432, + "claude_code_cost_usd": 3.7692859999999997 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/grader.json new file mode 100644 index 0000000..44460e2 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/grader.json @@ -0,0 +1,241 @@ +{ + "variant": "reference", + "judge_model": "claude-sonnet-5-5", + "suspicious_tokens": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 1297 + }, + "checks": { + "checks": [ + { + "name": "48_samples_to_odd_columns", + "pass": true, + "detail": "48 tubes -> 48 wells, odd columns only: True" + }, + { + "name": "step_order", + "pass": true, + "detail": "expected per well: beads > isopropanol > sample > remove > ethanol > remove > ethanol > remove > elution > recover" + }, + { + "name": "binding_volumes_40_250_250", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "mix_5x_after_sample", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "incubation_5min_before_magnet", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_4min_before_first_removal", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "supernatant_removed_each_step", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "magnet_engaged_for_all_removals", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "two_500ul_ethanol_washes", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "air_dry_4min", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_100ul", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "elution_off_magnet_then_90s_on", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "recover_70_100ul_one_well_each", + "pass": true, + "detail": "48/48 sample wells" + }, + { + "name": "distinct_elution_wells", + "pass": true, + "detail": "48 distinct elution wells" + }, + { + "name": "elution_plate_4C_before_recovery", + "pass": true, + "detail": "temperatures set: [(0.0, 4.0)]" + }, + { + "name": "fresh_tip_per_sample_no_cross_contact", + "pass": true, + "detail": "48 distinct tips for 48 sample transfers; violations: []" + }, + { + "name": "reservoir_columns_within_15ml", + "pass": false, + "detail": "uL drawn per reservoir column: 2: 1920, 4: 4800, 6: 6000, 7: 6000, 9: 12000, 10: 16000, 11: 12000, 12: 8000" + }, + { + "name": "recover_about_80ul", + "pass": true, + "detail": "70-90 uL of the 100 uL eluate, as the task's step 13 says; 48/48 sample wells" + } + ], + "checks_passed": 17, + "checks_total": 18, + "total_delay_min": 15.0, + "example_well": "A1", + "example_sequence": [ + "beads 40uL", + "isopropanol 250uL", + "sample 250uL", + "remove 540uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "ethanol 500uL (magnet)", + "remove 500uL (magnet)", + "elution 100uL", + "recover 80uL (magnet)" + ] + }, + "critical_failures": [ + "reservoir_columns_within_15ml" + ], + "judge": { + "items": [ + { + "id": "robot_practice", + "score": 0, + "evidence": "Failed check reservoir_columns_within_15ml: the ethanol split (etoh_cols[i%4] and [(i+1)%4]) draws 16000 uL from reservoir column 10, more than the 15 mL well holds.", + "votes": [ + 0, + 0, + 0 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "The run-log check fresh_tip_per_sample_no_cross_contact passed (48 distinct tips, no violations). Each p1000 sample tip and each p300 supernatant, wash-removal, elution and recovery tip is dropped per sample or column, and reagent-only tips never aspirate from sample wells.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "All 14 steps are done in order with the right volumes and delays (5 min, 4 min, 4 min dry, 30 s, 90 s). The magnet is engaged for removals, the temperature module is set to 4 C first, and 80 uL is recovered.", + "votes": [ + 1, + 1, + 0 + ] + }, + { + "id": "sample_handling", + "score": 1, + "evidence": "Check 48_samples_to_odd_columns passed, and each of the 48 tubes goes to its own well in columns 1,3,5,7,9,11 (dest = sample_cols[i//8][i%8]), with 48 distinct elution wells.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "binding_and_separation", + "score": 1, + "evidence": "The code adds 40 uL beads, then 250 uL isopropanol, then 250 uL sample with mix_after=(5,250). It then delays 5 min, engages the magnet for 4 min, and removes 2x270 uL to waste. Checks binding_volumes_40_250_250 and magnet_engaged_for_all_removals passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "washes_and_drying", + "score": 1, + "evidence": "Two washes each add 2x250 uL ethanol and remove 2x250 uL with the magnet engaged, then there is a 4 min delay before mag.disengage(). Checks two_500ul_ethanol_washes and air_dry_4min passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "elution_recovery", + "score": 1, + "evidence": "The code disengages the magnet, adds 100 uL elution buffer and mixes, waits 30 s, engages the magnet for 90 s, then transfers 80 uL to distinct wells of the 4 C elution plate.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 0.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "sample_handling": 1.0, + "binding_and_separation": 1.0, + "washes_and_drying": 1.0, + "elution_recovery": 1.0 + }, + "summary": "The protocol follows the task steps faithfully and passes tip handling, sample mapping, binding, washes and elution. It fails robot_practice because the ethanol split draws 16 mL from reservoir column 10, which holds 15 mL.", + "raw": "[{\"items\": [{\"id\": \"robot_practice\", \"score\": 0, \"evidence\": \"Failed check reservoir_columns_within_15ml: the ethanol split (etoh_cols[i%4] and [(i+1)%4]) draws 16000 uL from reservoir column 10, more than the 15 mL well holds.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The run-log check fresh_tip_per_sample_no_cross_contact passed (48 distinct tips, no violations). Each p1000 sample tip and each p300 supernatant, wash-removal, elution and recovery tip is dropped per sample or column, and reagent-only tips never aspirate from sample wells.\"}, {\"id\": \"fidelity_to_task\", \"score\": 1, \"evidence\": \"All 14 steps are done in order with the right volumes and delays (5 min, 4 min, 4 min dry, 30 s, 90 s). The magnet is engaged for removals, the temperature module is set to 4 C first, and 80 uL is recovered.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"Check 48_samples_to_odd_columns passed, and each of the 48 tubes goes to its own well in columns 1,3,5,7,9,11 (dest = sample_cols[i//8][i%8]), with 48 distinct elution wells.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"The code adds 40 uL beads, then 250 uL isopropanol, then 250 uL sample with mix_after=(5,250). It then delays 5 min, engages the magnet for 4 min, and removes 2x270 uL to waste. Checks binding_volumes_40_250_250 and magnet_engaged_for_all_removals passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two washes each add 2x250 uL ethanol and remove 2x250 uL with the magnet engaged, then there is a 4 min delay before mag.disengage(). Checks two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"The code disengages the magnet, adds 100 uL elution buffer and mixes, waits 30 s, engages the magnet for 90 s, then transfers 80 uL to distinct wells of the 4 C elution plate.\"}], \"summary\": \"The protocol follows the task steps faithfully and passes tip handling, sample mapping, binding, washes and elution. It fails robot_practice because the ethanol split draws 16 mL from reservoir column 10, which holds 15 mL.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 0, \"evidence\": \"The failed check reservoir_columns_within_15ml shows reservoir column 10 is asked to supply 16000 \\u00b5L, over the 15 mL well capacity. The cause is the uneven `etoh_cols[i % 4]` and `etoh_cols[(i + 1) % 4]` ethanol draw pattern.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The check fresh_tip_per_sample_no_cross_contact passed with 48 distinct tips. Every pick_up_tip is followed by drop_tip, and reagent-only tips never aspirate from sample wells. Fresh tips are used for supernatant removal, elution mixing and recovery.\"}, {\"id\": \"fidelity_to_task\", \"score\": 1, \"evidence\": \"All 14 steps are present in order with the specified volumes and delays (5 min, 4 min, 4 min dry, 30 s, 90 s). Supernatant is removed as 2\\u00d7270 \\u00b5L, 80 \\u00b5L is recovered, and the temperature module is set to 4 \\u00b0C first. I found no deviation outside the reservoir over-draw already penalised under robot_practice.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"The check 48_samples_to_odd_columns passed. Tubes are mapped one-to-one into sample_cols (odd columns) and each is recovered into its own elution well (distinct_elution_wells passed).\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"The code adds beads 40, then isopropanol 250, then sample 250 with mix_after=(5,250). It then delays 5 min, engages the magnet for 4 min, and removes 2\\u00d7270 \\u00b5L with the magnet on.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two washes of 2\\u00d7250 \\u00b5L ethanol, each removed with 2\\u00d7250 \\u00b5L while the magnet is engaged. A 4 min delay with the magnet engaged follows, then mag.disengage().\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"100 \\u00b5L elution buffer is added after mag.disengage() and mixed 5 times. The code waits 30 s, engages the magnet, waits 90 s, then transfers 80 \\u00b5L to distinct wells (recover_about_80ul passed).\"}], \"summary\": \"The protocol follows the specified steps with correct volumes, timings, magnet states, tip handling and an 80 \\u00b5L recovery. It fails robot_practice because the ethanol is spread unevenly across reservoir columns 9\\u201312, which asks column 10 for 16 mL against a 15 mL capacity.\"}, {\"items\": [{\"id\": \"robot_practice\", \"score\": 0, \"evidence\": \"reservoir_columns_within_15ml failed: reservoir column 10 is drawn 16000 uL (>15 mL), because ethanol is split as etoh_cols[i%4] and etoh_cols[(i+1)%4] unevenly.\"}, {\"id\": \"tips_and_contamination\", \"score\": 1, \"evidence\": \"The p1000 takes a fresh tip per sample and the multi takes a fresh tip per column for every aspirate from a sample well. The shared reagent-dispense tips never aspirate from sample wells, every tip is dropped, and fresh_tip_per_sample_no_cross_contact passed.\"}, {\"id\": \"fidelity_to_task\", \"score\": 0, \"evidence\": \"The ethanol distribution draws 16000 uL from reservoir column 10, which exceeds its 15 mL capacity and is not a faithful, feasible execution of the specified deck.\"}, {\"id\": \"sample_handling\", \"score\": 1, \"evidence\": \"48_samples_to_odd_columns passed, and distinct_elution_wells shows 48 distinct elution wells, with samples going from tubes to sample_cols[i//8][i%8] in the odd columns.\"}, {\"id\": \"binding_and_separation\", \"score\": 1, \"evidence\": \"The code adds beads 40, then isopropanol 250, then sample 250 with mix_after=(5,250). It then delays 5 min, engages the magnet for 4 min, and removes 2x270 uL supernatant to waste; the matching checks passed.\"}, {\"id\": \"washes_and_drying\", \"score\": 1, \"evidence\": \"Two washes of 2x250 uL ethanol are each removed with the magnet engaged, followed by a 4 min delay with the magnet on and then mag.disengage() before elution; two_500ul_ethanol_washes and air_dry_4min passed.\"}, {\"id\": \"elution_recovery\", \"score\": 1, \"evidence\": \"The code adds 100 uL elution buffer after disengaging and mixes 5x. It then waits 30 s, engages for 90 s, and transfers 80 uL to distinct wells of the 4 C elution plate; recover_about_80ul passed.\"}], \"summary\": \"The protocol follows the extraction steps correctly, with correct volumes, magnet use, timings and tip handling. However, it draws 16000 uL from reservoir column 10, which exceeds the 15 mL well capacity, so robot_practice and fidelity_to_task fail.\"}]", + "level": "easy", + "score": 0.75, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "sample_handling": 0.0625, + "binding_and_separation": 0.0625, + "washes_and_drying": 0.0625, + "elution_recovery": 0.0625 + }, + "model": "anthropic/claude-sonnet-5.5", + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9444, + "judge_score": 0.75, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 1.0, + "deterministic_reward": 0.4722, + "rubric_robot_practice": 0.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/protocol.py new file mode 100644 index 0000000..849024c --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/protocol.py @@ -0,0 +1,157 @@ +""" +Automated low-cost SARS-CoV-2 RNA extraction (in-house magnetic-bead protocol) +adapted from PLOS ONE 2021, doi:10.1371/journal.pone.0246302. + +48 samples, processed in the odd columns (1, 3, 5, 7, 9, 11) of a deep-well +plate on the Magnetic Module (slot 4). Eluates are recovered into the same +columns of a 200 uL plate on the Temperature Module (slot 6) held at 4 degC. + +Pipettes: + left - p1000_single_gen2 (sample transfer from tubes, 1 fresh tip / sample) + right - p300_multi_gen2 (all plate-column work) + +Tip discipline: any tip that aspirates from a sample well is used on that +column only (with the multi, each of the 8 tips touches exactly one well). +Reagent-dispensing tips are only reused while they never aspirate from a +sample well. +""" + +metadata = { + 'apiLevel': '2.11', + 'protocolName': 'OT-2 magnetic-bead RNA extraction, 48 samples', + 'description': 'In-house bead-based SARS-CoV-2 RNA extraction (PLOS ONE ' + '2021, doi:10.1371/journal.pone.0246302), 48 samples.', +} + +# Odd plate columns (0-based column indices 0,2,4,6,8,10 == columns 1,3,5,7,9,11) +SAMPLE_COL_IDX = [0, 2, 4, 6, 8, 10] + + +def run(protocol): + # ------------------------------------------------------------------ labware + waste = protocol.load_labware('usascientific_96_wellplate_2.4ml_deep', '1', + 'waste plate') + tips200 = [protocol.load_labware('opentrons_96_filtertiprack_200ul', s) + for s in ['2', '3', '9']] + tips1000 = protocol.load_labware('opentrons_96_filtertiprack_1000ul', '11') + + mag = protocol.load_module('magnetic module', '4') + mag_plate = mag.load_labware('usascientific_96_wellplate_2.4ml_deep', + label='sample/extraction plate') + + temp = protocol.load_module('tempdeck', '6') + elution_plate = temp.load_labware('thermo_96_wellplate_200ul', + label='elution plate') + + reservoir = protocol.load_labware('nest_12_reservoir_15ml', '5', + 'reagent reservoir') + rack10 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '10', + 'samples 1-24') + rack7 = protocol.load_labware( + 'opentrons_24_tuberack_eppendorf_2ml_safelock_snapcap', '7', + 'samples 25-48') + + # ---------------------------------------------------------------- pipettes + p1000 = protocol.load_instrument('p1000_single_gen2', 'left', + tip_racks=[tips1000]) + p300m = protocol.load_instrument('p300_multi_gen2', 'right', + tip_racks=tips200) + + # ------------------------------------------------------------- reagents + beads = reservoir.columns()[1] # reservoir column 2 + elution_buf = reservoir.columns()[3] # reservoir column 4 + iso_cols = [reservoir.columns()[5], reservoir.columns()[6]] # columns 6-7 + etoh_cols = reservoir.columns()[8:12] # columns 9-12 + + sample_cols = [mag_plate.columns()[i] for i in SAMPLE_COL_IDX] + waste_cols = [waste.columns()[i] for i in SAMPLE_COL_IDX] + elution_cols = [elution_plate.columns()[i] for i in SAMPLE_COL_IDX] + + # Sample tube order: column-major over the two 24-position racks. + # Samples 1-24 -> rack in slot 10 -> plate columns 1, 3, 5 + # Samples 25-48 -> rack in slot 7 -> plate columns 7, 9, 11 + tubes = rack10.wells()[:24] + rack7.wells()[:24] + + # -------------------------------------------------- step 1: initial state + mag.disengage() + temp.set_temperature(4) + + # ------------------------------------- steps 2-4: beads, isopropanol, sample + # Step 2: 40 uL magnetic beads into every sample well (dispense-only tip). + p300m.pick_up_tip() + for col in sample_cols: + p300m.transfer(40, beads, col, new_tip='never') + p300m.drop_tip() + + # Step 3: 250 uL isopropanol into every sample well (dispense-only tip). + p300m.pick_up_tip() + for i, col in enumerate(sample_cols): + p300m.transfer(250, iso_cols[i % 2], col, new_tip='never') + p300m.drop_tip() + + # Step 4: 250 uL of each sample from its tube into its own well, + # fresh tip per sample, mix by pipetting 5 times. + for i, tube in enumerate(tubes): + dest = sample_cols[i // 8][i % 8] # column-major within odd columns + p1000.pick_up_tip() + p1000.transfer(250, tube, dest, new_tip='never', mix_after=(5, 250)) + p1000.drop_tip() + + # ----------------------------------------- step 5: incubate 5 min at RT + protocol.delay(minutes=5, msg='Incubating 5 min at room temperature') + + # ------------------------------------------- step 6: engage magnet, 4 min + mag.engage() + protocol.delay(minutes=4, msg='Incubating 4 min on magnet') + + # ----------------------------------------- step 7: remove supernatant + for i, col in enumerate(sample_cols): + p300m.pick_up_tip() + # ~540 uL total, in two 270 uL aspirates (p300 limit is 300 uL) + p300m.transfer(270, col, waste_cols[i], new_tip='never') + p300m.transfer(270, col, waste_cols[i], new_tip='never') + p300m.drop_tip() + + # -------------------------------------- steps 8-9: two 70% ethanol washes + for wash in range(2): + # Dispense 500 uL ethanol (2 x 250 uL) into every sample well, + # magnet stays engaged; dispense-only tip reused across columns. + p300m.pick_up_tip() + for i, col in enumerate(sample_cols): + p300m.transfer(250, etoh_cols[i % 4], col, new_tip='never') + p300m.transfer(250, etoh_cols[(i + 1) % 4], col, new_tip='never') + p300m.drop_tip() + + # Remove ethanol with the magnet engaged (fresh tips per column). + for i, col in enumerate(sample_cols): + p300m.pick_up_tip() + p300m.transfer(250, col, waste_cols[i], new_tip='never') + p300m.transfer(250, col, waste_cols[i], new_tip='never') + p300m.drop_tip() + + # -------------------------------------- step 10: air-dry beads 4 min + protocol.delay(minutes=4, msg='Air-drying beads 4 min, magnet engaged') + + # --------------------------- step 11: disengage, resuspend in elution buf + mag.disengage() + for col in sample_cols: + # Fresh tips per column: this tip mixes (aspirates) in the sample wells. + p300m.pick_up_tip() + p300m.transfer(100, elution_buf, col, new_tip='never') + p300m.mix(5, 100, col[0]) + p300m.drop_tip() + + # ------------------------- step 12: wait 30 s, engage magnet, wait 90 s + protocol.delay(seconds=30, msg='Waiting 30 s off magnet') + mag.engage() + protocol.delay(seconds=90, msg='Waiting 90 s on magnet') + + # ---------------- step 13: transfer 80 uL eluate to the 4 degC elution plate + for i, col in enumerate(sample_cols): + p300m.pick_up_tip() + p300m.transfer(80, col, elution_cols[i], new_tip='never') + p300m.drop_tip() + + # ---------------------------------------------- step 14: disengage magnet + mag.disengage() diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/reward.json new file mode 100644 index 0000000..341c92f --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/reward.json @@ -0,0 +1 @@ +{"reward": 0.3, "sim_pass": 1.0, "checks_frac": 0.9444, "judge_score": 0.75, "suspicious_code": 0.0, "judge_error": 0.0, "hack_detected": 0.0, "critical_fail": 1.0, "deterministic_reward": 0.4722, "rubric_robot_practice": 0.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_sample_handling": 1.0, "rubric_binding_and_separation": 1.0, "rubric_washes_and_drying": 1.0, "rubric_elution_recovery": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/trial.json new file mode 100644 index 0000000..fda44e3 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/opentrons-rna-extraction/trial.json @@ -0,0 +1,32 @@ +{ + "task": "opentrons-rna-extraction", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 0.3, + "sim_pass": 1.0, + "checks_frac": 0.9444, + "judge_score": 0.75, + "suspicious_code": 0.0, + "judge_error": 0.0, + "hack_detected": 0.0, + "critical_fail": 1.0, + "deterministic_reward": 0.4722, + "rubric_robot_practice": 0.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_sample_handling": 1.0, + "rubric_binding_and_separation": 1.0, + "rubric_washes_and_drying": 1.0, + "rubric_elution_recovery": 1.0 + }, + "exception": null, + "cost_usd": 0.215678, + "input_tokens": 328584, + "output_tokens": 14413, + "started_at": "2026-10-07T17:52:54.112795Z", + "finished_at": "2026-10-07T18:10:20.532445Z", + "cache_tokens": 301696, + "claude_code_cost_usd": 0.645613 +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/grader.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/grader.json new file mode 100644 index 0000000..4d5eaad --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/grader.json @@ -0,0 +1,137 @@ +{ + "lint": [], + "traps": [], + "simulation": { + "ok": true, + "n_commands": 8 + }, + "not_applicable": [], + "checks": [ + { + "name": "simulator_ran", + "pass": true, + "detail": "" + }, + { + "name": "tip_before_aspirate", + "pass": true, + "detail": "[]" + }, + { + "name": "no_overdispense", + "pass": true, + "detail": "[]" + }, + { + "name": "no_aspirate_from_empty_well", + "pass": true, + "detail": "[]" + }, + { + "name": "tip_dropped_at_end", + "pass": true, + "detail": "" + }, + { + "name": "no_cross_contamination", + "pass": true, + "detail": "[]" + }, + { + "name": "end_state:plate", + "pass": true, + "detail": "expected volumes [100.0, 100.0], got [100.0, 100.0] (positions free)" + } + ], + "judge": { + "model": "anthropic/claude-sonnet-5.5", + "items": [ + { + "id": "robot_practice", + "score": 1, + "evidence": "apiLevel 2.15 is valid, and the p300 handles 100 \u00b5L (about 220 \u00b5L aspirated including the default disposal volume, under 300). The simulator checks all passed.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "tips_and_contamination", + "score": 1, + "evidence": "The checks tip_before_aspirate, tip_dropped_at_end and no_cross_contamination passed. distribute with new_tip='always' picks up one tip and drops it at the end. The has_tip guard is a harmless extra.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "fidelity_to_task", + "score": 1, + "evidence": "The code moves 100 \u00b5L from reservoir A1 to plate A1 and B1 with the p300, and end_state:plate shows [100, 100]. The extra disposal-volume aspirate is blown out and does not change the outcome.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "deck_and_hardware", + "score": 1, + "evidence": "Reservoir is in slot 1 and plate in slot 2 with the required labels. Tip racks are in slots 10 and 11, with p20 on left and p300 on right.", + "votes": [ + 1, + 1, + 1 + ] + }, + { + "id": "volumes_and_wells", + "score": 1, + "evidence": "p300.distribute(100, reservoir A1, [plate A1, plate B1]) puts exactly 100 \u00b5L in each target well and touches no other wells.", + "votes": [ + 1, + 1, + 1 + ] + } + ], + "scores": { + "robot_practice": 1.0, + "tips_and_contamination": 1.0, + "fidelity_to_task": 1.0, + "deck_and_hardware": 1.0, + "volumes_and_wells": 1.0 + }, + "summary": "The protocol simulates cleanly, and all measured checks pass. It uses the p300 with distribute to put 100 \u00b5L into each of plate A1 and B1, with correct labware, slots and tip handling.", + "level": "easy", + "score": 1.0, + "weights": { + "robot_practice": 0.25, + "tips_and_contamination": 0.25, + "fidelity_to_task": 0.25, + "deck_and_hardware": 0.125, + "volumes_and_wells": 0.125 + }, + "provider": "openrouter", + "votes": 3, + "failed_votes": 0 + }, + "critical_failures": [], + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + } +} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/protocol.py b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/protocol.py new file mode 100644 index 0000000..4875490 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/protocol.py @@ -0,0 +1,34 @@ +metadata = { + 'protocolName': 'Split 200 uL into two 100 uL wells', + 'apiLevel': '2.15', +} + + +def run(protocol): + # Load labware exactly as the operator set up the deck + reservoir = protocol.load_labware( + 'nest_1_reservoir_195ml', 1, label='reservoir') + plate = protocol.load_labware( + 'corning_96_wellplate_360ul_flat', 2, label='plate') + + # Load tip racks + tips20 = protocol.load_labware('opentrons_96_tiprack_20ul', 10) + tips300 = protocol.load_labware('opentrons_96_tiprack_300ul', 11) + + # Pipettes + p20 = protocol.load_instrument('p20_single_gen2', 'left', tip_racks=[tips20]) + p300 = protocol.load_instrument('p300_single_gen2', 'right', tip_racks=[tips300]) + + # 100 uL each into plate wells A1 and B1 from reservoir A1 (100 uL is within + # the p300's 20-300 uL range) + p300.distribute( + 100, + reservoir.wells_by_name()['A1'], + [plate.wells_by_name()['A1'], plate.wells_by_name()['B1']], + new_tip='always', + ) + + # Make sure no tip is left on the pipette at the end + if p300.has_tip: + p300.drop_tip() + p300.reset_tipracks() diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/reward.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/reward.json new file mode 100644 index 0000000..5b16841 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/reward.json @@ -0,0 +1 @@ +{"reward": 1.0, "sim_pass": 1.0, "checks_frac": 1.0, "lint_violations": 0.0, "hack_detected": 0.0, "deterministic_reward": 1.0, "judge_score": 1.0, "critical_fail": 0.0, "rubric_robot_practice": 1.0, "rubric_tips_and_contamination": 1.0, "rubric_fidelity_to_task": 1.0, "rubric_deck_and_hardware": 1.0, "rubric_volumes_and_wells": 1.0} diff --git a/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/trial.json b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/trial.json new file mode 100644 index 0000000..83e1932 --- /dev/null +++ b/results/runs/2026-10-07-openrouter/qwen3.8-2.4t-a95b/split-200ul-two-wells/trial.json @@ -0,0 +1,29 @@ +{ + "task": "split-200ul-two-wells", + "model": "qwen/qwen3.8-2.4t-a95b", + "agent": "claude-code", + "via": "openrouter", + "rewards": { + "reward": 1.0, + "sim_pass": 1.0, + "checks_frac": 1.0, + "lint_violations": 0.0, + "hack_detected": 0.0, + "deterministic_reward": 1.0, + "judge_score": 1.0, + "critical_fail": 0.0, + "rubric_robot_practice": 1.0, + "rubric_tips_and_contamination": 1.0, + "rubric_fidelity_to_task": 1.0, + "rubric_deck_and_hardware": 1.0, + "rubric_volumes_and_wells": 1.0 + }, + "exception": null, + "cost_usd": 0.02959, + "input_tokens": 49406, + "output_tokens": 1007, + "started_at": "2026-10-07T18:10:20.587132Z", + "finished_at": "2026-10-07T18:11:31.810271Z", + "cache_tokens": 43008, + "claude_code_cost_usd": 0.07866899999999999 +} diff --git a/scripts/benchmark_report.py b/scripts/benchmark_report.py index c311afb..2699e89 100644 --- a/scripts/benchmark_report.py +++ b/scripts/benchmark_report.py @@ -17,6 +17,7 @@ "simulator_ran": "protocol runs in the simulator", "deck_labware": "right labware, label and slot (fixed deck)", "end_state": "end-state volumes match the IR ground truth", + "composition": "where the paper fixes no exact volumes: every well gets each input, equal volumes, in the paper's range", "tip_before_aspirate": "never pipettes without a tip", "no_overdispense": "never dispenses more than it holds", "no_aspirate_from_empty_well": "never aspirates from an empty well", diff --git a/scripts/build_site.py b/scripts/build_site.py index 2d77af2..ddf5f51 100644 --- a/scripts/build_site.py +++ b/scripts/build_site.py @@ -16,9 +16,9 @@ ROOT = pathlib.Path(__file__).resolve().parent.parent REPO = "https://github.com/PhysicalAIBenchmarks/Text2WetLab" PAGES = ["index.html", "plr_coverage_table.html", "plr_embodiment_pairs.html", "docs/demo.html", "docs/preprint/preprint.html"] -ORDER = ["claude-sonnet-5.5", "claude-opus-5.5", "claude-fable-5.1", "gpt-6.1-sol"] +ORDER = ["claude-sonnet-5.5", "claude-opus-5.5", "claude-fable-5.1", "gpt-6.1-sol", "qwen3.8-2.4t-a95b", "deepseek-v4-pro-0813"] NAME = {"claude-sonnet-5.5": "Claude Sonnet 5.5", "claude-opus-5.5": "Claude Opus 5.5", "claude-fable-5.1": "Claude Fable 5.1", - "gpt-6.1-sol": "GPT-6.1 Sol"} + "gpt-6.1-sol": "GPT-6.1 Sol", "qwen3.8-2.4t-a95b": "Qwen3.8-2.4T-A95B", "deepseek-v4-pro-0813": "DeepSeek V4 Pro"} LOCAL_REF = re.compile(r'(?:src|href|poster|data-src)=["\']([^"\'#?]+)') diff --git a/scripts/compare_jobs.py b/scripts/compare_jobs.py new file mode 100644 index 0000000..c2475bb --- /dev/null +++ b/scripts/compare_jobs.py @@ -0,0 +1,84 @@ +"""Compare Harbor jobs: wall-clock time, where the time goes per trial, cost and reward, task by task. + + python scripts/compare_jobs.py LABEL=JOB_DIR [LABEL=JOB_DIR ...] [--prices OPENROUTER_MODELS_JSON] + +Use it to compare backends (local Docker vs Modal) for the same model, or repeated runs of one model (run-to-run +variation at pass@1). Times come from each trial's result.json; a job's wall clock runs from its first trial's start to +its last trial's finish. Rewards are as graded at run time (not any later regrade). With --prices, non-Claude models' +cost comes from their tokens at OpenRouter prices (Claude Code prices every model as Claude; see export_run.py). +""" +import json +import pathlib +import statistics +import sys +from datetime import datetime + +from export_run import openrouter_cost + + +def ts(s: str | None) -> datetime | None: + return datetime.fromisoformat(s.replace("Z", "+00:00")) if s else None + + +def span(d: dict | None) -> float | None: + if not d or not d.get("started_at") or not d.get("finished_at"): + return None + return (ts(d["finished_at"]) - ts(d["started_at"])).total_seconds() + + +PRICES: dict = {} + + +def load(job: pathlib.Path) -> dict: + trials = {} + for f in sorted(job.glob("*/result.json")): + t = json.loads(f.read_text()) + r = (t.get("verifier_result") or {}).get("rewards") or {} + agent, model = t.get("agent_result") or {}, t["config"]["agent"]["model_name"] + cost = (openrouter_cost(agent, PRICES[model]) if agent and model in PRICES and not model.startswith("anthropic/") + else agent.get("cost_usd") or 0.0) + trials[t["task_name"].split("/")[-1]] = { + "reward": r.get("reward"), "exception": (t.get("exception_info") or {}).get("exception_type"), + "cost": cost, + "start": ts(t.get("started_at")), "end": ts(t.get("finished_at")), "total": span(t), + "env": span(t.get("environment_setup")), "agent": span(t.get("agent_execution")), "verify": span(t.get("verifier"))} + return trials + + +def fmt_s(v: float | None) -> str: + return "-" if v is None else f"{v / 60:.1f} min" if v >= 90 else f"{v:.0f} s" + + +def main() -> int: + jobs, args = {}, sys.argv[1:] + if "--prices" in args: + i = args.index("--prices") + PRICES.update({m["id"]: m["pricing"] for m in json.loads(pathlib.Path(args[i + 1]).read_text())["data"]}) + del args[i:i + 2] + for arg in args: + label, _, path = arg.partition("=") + jobs[label] = load(pathlib.Path(path).expanduser()) + print("| Job | Trials | Wall clock | Median trial | Median env setup | Median agent | Median verifier | Agent cost | Mean reward (answered) |") + print("|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|") + for label, tr in jobs.items(): + done = [t for t in tr.values() if t["end"]] + wall = (max(t["end"] for t in done) - min(t["start"] for t in done)).total_seconds() if done else None + med = lambda k: statistics.median([t[k] for t in tr.values() if t[k] is not None]) if any(t[k] is not None for t in tr.values()) else None + answered = [t["reward"] for t in tr.values() if t["reward"] is not None and not t["exception"]] + score = f"{statistics.mean(answered):.3f} (n={len(answered)})" if answered else "-" + print(f"| {label} | {len(tr)} | {fmt_s(wall)} | {fmt_s(med('total'))} | {fmt_s(med('env'))} | {fmt_s(med('agent'))} | " + f"{fmt_s(med('verify'))} | ${sum(t['cost'] for t in tr.values()):.2f} | {score} |") + tasks = sorted({k for tr in jobs.values() for k in tr}, key=lambda k: (k.endswith("-hard"), k)) + labels = list(jobs) + print("\n| Task | " + " | ".join(labels) + " |\n|---|" + ":---:|" * len(labels)) + for k in tasks: + cells = [] + for label in labels: + t = jobs[label].get(k) + cells.append("-" if not t else t["exception"] or ("-" if t["reward"] is None else f"{t['reward']:.2f}")) + print(f"| `{k}` | " + " | ".join(cells) + " |") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/export_run.py b/scripts/export_run.py index 72f4339..21c16dd 100644 --- a/scripts/export_run.py +++ b/scripts/export_run.py @@ -6,6 +6,10 @@ graded), reward.json and grader.json (lint, traps, simulator summary, every check, the judge's verdicts). With --regraded (the --out folder of scripts/regrade_jobs.py) a regraded trial's reward and grader record replace the originals, and the original rewards are kept in trial.json as original_rewards. Transcripts and logs are not copied. + +Claude Code prices every model as a Claude model, so its cost_usd is wrong for any other model. With --prices (a saved +copy of https://openrouter.ai/api/v1/models) cost_usd is recomputed from the token counts at that model's OpenRouter +prices, and Claude Code's figure is kept as claude_code_cost_usd. The run's prices are saved next to it as prices.json. """ import argparse import json @@ -17,13 +21,24 @@ def record(verifier: pathlib.Path) -> dict: return next((json.loads((verifier / n).read_text()) for n in ("result.json", "judge.json") if (verifier / n).exists()), {}) +def openrouter_cost(agent: dict, pricing: dict) -> float: + """Uncached input at the prompt price, cached input at the cache-read price, output at the completion price.""" + cached = agent.get("n_cache_tokens") or 0 + uncached = (agent.get("n_input_tokens") or 0) - cached + return round(uncached * float(pricing["prompt"]) + cached * float(pricing.get("input_cache_read") or pricing["prompt"]) + + (agent.get("n_output_tokens") or 0) * float(pricing["completion"]), 6) + + def main(): ap = argparse.ArgumentParser() ap.add_argument("dest", type=pathlib.Path) ap.add_argument("jobs", nargs="+", type=pathlib.Path) ap.add_argument("--regraded", type=pathlib.Path) ap.add_argument("--via", default="openrouter") + ap.add_argument("--prices", type=pathlib.Path, help="saved OpenRouter /api/v1/models response") a = ap.parse_args() + prices = {m["id"]: m["pricing"] for m in json.loads(a.prices.read_text())["data"]} if a.prices else {} + used = {} for job in a.jobs: for f in sorted(job.glob("*/result.json")): t = json.loads(f.read_text()) @@ -39,7 +54,11 @@ def main(): "rewards": rewards, "exception": (t.get("exception_info") or {}).get("exception_type"), "cost_usd": agent.get("cost_usd"), "input_tokens": agent.get("n_input_tokens"), "output_tokens": agent.get("n_output_tokens"), "started_at": t.get("started_at"), - "finished_at": t.get("finished_at")} + "finished_at": t.get("finished_at"), "cache_tokens": agent.get("n_cache_tokens")} + if not model_name.startswith("anthropic/") and model_name in prices and agent: + trial["claude_code_cost_usd"] = trial["cost_usd"] + trial["cost_usd"] = openrouter_cost(agent, prices[model_name]) + used[model_name] = prices[model_name] verifier = f.parent / "verifier" grader = record(verifier) regraded = a.regraded / job.name / task if a.regraded else None @@ -56,6 +75,10 @@ def main(): shutil.copy(verifier / "protocol.py", out / "protocol.py") print(f"{model_name:32} {task:40} {trial['rewards'] and trial['rewards'].get('reward')}" + (f" (was {rewards.get('reward')})" if "original_rewards" in trial and rewards else "")) + if used: + saved = a.dest / "prices.json" + old = json.loads(saved.read_text()) if saved.exists() else {} + saved.write_text(json.dumps({**old, **used}, indent=1, sort_keys=True) + "\n") if __name__ == "__main__": diff --git a/scripts/harbor_adversarial.py b/scripts/harbor_adversarial.py index 04b5b14..aa58a5b 100644 --- a/scripts/harbor_adversarial.py +++ b/scripts/harbor_adversarial.py @@ -17,6 +17,8 @@ import tempfile ROOT = pathlib.Path(__file__).resolve().parent.parent +# attacks whose result is a valid protocol for that task, so full marks are right +VALID = {("colony-pcr-screening-hard", "halve_volumes"): "9 uL mix + 0.5 + 0.5 = 10 uL, the paper's OT-2 reaction volume"} OT_VENV = pathlib.Path(os.environ.get("OT_VENV", ROOT / ".venv-ot")) @@ -116,12 +118,14 @@ def main(): continue results[task.name][name] = run_one(task, name, mutated, pathlib.Path(data_for.get(task.name, h / "environment/data"))) r = results[task.name][name] + if (task.name, name) in VALID: + r["valid"] = VALID[(task.name, name)] print(f"{task.name:34} {name:36} reward={r['reward']} {'| ' + ','.join(r.get('failed_checks', []))[:70] if r.get('failed_checks') else ''}{' | lint:' + r['lint'][0][:40] if r.get('lint') else ''}", flush=True) pathlib.Path(a.out).parent.mkdir(parents=True, exist_ok=True) pathlib.Path(a.out).write_text(json.dumps(results, indent=1)) - holes = [(t, n) for t, rs in results.items() for n, r in rs.items() if not n.startswith("control") and r.get("reward") == 1.0] - broken = [(t, n) for t, rs in results.items() for n, r in rs.items() if n.startswith("control") and r.get("reward") != 1.0] - print(f"\nHOLES (attack scored 1.0): {len(holes)} {holes}\nCONTROL FAILURES: {len(broken)} {broken}") + holes = [(t, n) for t, rs in results.items() for n, r in rs.items() if not n.startswith("control") and r.get("reward") == 1.0 and not r.get("valid")] + broken = [(t, n) for t, rs in results.items() for n, r in rs.items() if (n.startswith("control") or r.get("valid")) and r.get("reward") != 1.0] + print(f"\nHOLES (attack scored 1.0): {len(holes)} {holes}\nCONTROL FAILURES (incl. valid variants): {len(broken)} {broken}") if __name__ == "__main__": diff --git a/scripts/make_harbor.py b/scripts/make_harbor.py index e646917..5820b3b 100644 --- a/scripts/make_harbor.py +++ b/scripts/make_harbor.py @@ -173,7 +173,8 @@ def checks_json(folder: pathlib.Path) -> str: cfg = tomllib.loads((folder / "task.toml").read_text()).get("checks", {}) out = {"free_wells": sorted(cfg.get("free_wells", []))} if cfg.get("not_from_paper"): - out["not_from_paper"] = sorted(cfg["not_from_paper"]) + nfp = cfg["not_from_paper"] # [containers], or {container: [min uL, max uL] the paper supports} + out["not_from_paper"] = dict(sorted(nfp.items())) if isinstance(nfp, dict) else sorted(nfp) return json.dumps(out) + "\n" diff --git a/scripts/regrade_jobs.py b/scripts/regrade_jobs.py index 2088edc..dd18448 100644 --- a/scripts/regrade_jobs.py +++ b/scripts/regrade_jobs.py @@ -1,6 +1,7 @@ """Regrade saved Harbor trials with the current graders, without re-running any agent. python scripts/regrade_jobs.py JOB_DIR [JOB_DIR ...] [--judge [--all]] [--data TASK=DIR ...] [--out DIR] [--workers N] + [--only results/runs//PROVISIONAL.json] Each trial's graded protocol (verifier/protocol.py) goes back through tasks//tests/grade.py outside Docker. Without --judge only the deterministic layers run (SKIP_JUDGE=1, free). With --judge the LLM judge runs too, but only for @@ -45,9 +46,17 @@ def main(): ap.add_argument("--data", nargs="*", default=[]) ap.add_argument("--out", type=pathlib.Path) ap.add_argument("--workers", type=int, default=4, help="trials graded at once") + ap.add_argument("--only", type=pathlib.Path, help="PROVISIONAL.json of a run: regrade only the / it lists") a = ap.parse_args() data_for = dict(x.split("=", 1) for x in a.data) trials = [(job, f) for job in a.jobs for f in sorted(job.glob("*/result.json"))] + if a.only: + wanted = {k for k in json.loads(a.only.read_text()) if not k.startswith("_")} + def key(f): + t = json.loads(f.read_text()) + return f'{t["config"]["agent"]["model_name"].split("/")[-1]}/{t["task_name"].split("/")[-1]}' + trials = [(job, f) for job, f in trials if key(f) in wanted] + print(f"regrading {len(trials)} of the {len(wanted)} listed trials", file=sys.stderr) def regrade(job_and_file) -> dict: job, f = job_and_file diff --git a/tasks/a1-a12-100ul/tests/grade.py b/tasks/a1-a12-100ul/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/a1-a12-100ul/tests/grade.py +++ b/tasks/a1-a12-100ul/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/a1-a12-100ul/tests/protocol_lint.py b/tasks/a1-a12-100ul/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/a1-a12-100ul/tests/protocol_lint.py +++ b/tasks/a1-a12-100ul/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/a1-a12-100ul/tests/spec_check.py b/tasks/a1-a12-100ul/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/a1-a12-100ul/tests/spec_check.py +++ b/tasks/a1-a12-100ul/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/ampure-bead-cleanup/tests/grade.py b/tasks/ampure-bead-cleanup/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/ampure-bead-cleanup/tests/grade.py +++ b/tasks/ampure-bead-cleanup/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/ampure-bead-cleanup/tests/protocol_lint.py b/tasks/ampure-bead-cleanup/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/ampure-bead-cleanup/tests/protocol_lint.py +++ b/tasks/ampure-bead-cleanup/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/ampure-bead-cleanup/tests/spec_check.py b/tasks/ampure-bead-cleanup/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/ampure-bead-cleanup/tests/spec_check.py +++ b/tasks/ampure-bead-cleanup/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/colony-pcr-screening-hard/task.toml b/tasks/colony-pcr-screening-hard/task.toml index 9146938..cc0c770 100644 --- a/tasks/colony-pcr-screening-hard/task.toml +++ b/tasks/colony-pcr-screening-hard/task.toml @@ -34,8 +34,10 @@ gpus = 0 [checks] free_wells = [] # The ground truth (from the easy task) is a 20 uL reaction; the paper's OT-2 workflow is 10 uL and the deck gives no -# primer concentration, so the paper does not determine the PCR plate's volumes. The rubric judges the reaction. -not_from_paper = ["pcr_plate"] +# primer concentration, so the paper does not determine the PCR plate's exact volumes. What it does fix is checked +# instead: every well gets master mix, its primers and its colony, all wells alike, and a reaction no smaller than the +# paper's OT-2 one (9 uL mix + 1 uL colony = 10 uL) nor larger than a standard Q5 reaction (25 uL). The rubric judges the rest. +not_from_paper = { pcr_plate = [10, 25] } [[metadata.papers]] slug = "slowpoke" diff --git a/tasks/colony-pcr-screening-hard/tests/checks.json b/tasks/colony-pcr-screening-hard/tests/checks.json index 1950f50..07baf39 100644 --- a/tasks/colony-pcr-screening-hard/tests/checks.json +++ b/tasks/colony-pcr-screening-hard/tests/checks.json @@ -1 +1 @@ -{"free_wells": [], "not_from_paper": ["pcr_plate"]} +{"free_wells": [], "not_from_paper": {"pcr_plate": [10, 25]}} diff --git a/tasks/colony-pcr-screening-hard/tests/grade.py b/tasks/colony-pcr-screening-hard/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/colony-pcr-screening-hard/tests/grade.py +++ b/tasks/colony-pcr-screening-hard/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/colony-pcr-screening-hard/tests/protocol_lint.py b/tasks/colony-pcr-screening-hard/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/colony-pcr-screening-hard/tests/protocol_lint.py +++ b/tasks/colony-pcr-screening-hard/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/colony-pcr-screening-hard/tests/spec_check.py b/tasks/colony-pcr-screening-hard/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/colony-pcr-screening-hard/tests/spec_check.py +++ b/tasks/colony-pcr-screening-hard/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/colony-pcr-screening/tests/grade.py b/tasks/colony-pcr-screening/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/colony-pcr-screening/tests/grade.py +++ b/tasks/colony-pcr-screening/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/colony-pcr-screening/tests/protocol_lint.py b/tasks/colony-pcr-screening/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/colony-pcr-screening/tests/protocol_lint.py +++ b/tasks/colony-pcr-screening/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/colony-pcr-screening/tests/spec_check.py b/tasks/colony-pcr-screening/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/colony-pcr-screening/tests/spec_check.py +++ b/tasks/colony-pcr-screening/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/ecoli-heat-shock-transformation-hard/tests/grade.py b/tasks/ecoli-heat-shock-transformation-hard/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/ecoli-heat-shock-transformation-hard/tests/grade.py +++ b/tasks/ecoli-heat-shock-transformation-hard/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/ecoli-heat-shock-transformation-hard/tests/protocol_lint.py b/tasks/ecoli-heat-shock-transformation-hard/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/ecoli-heat-shock-transformation-hard/tests/protocol_lint.py +++ b/tasks/ecoli-heat-shock-transformation-hard/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/ecoli-heat-shock-transformation-hard/tests/spec_check.py b/tasks/ecoli-heat-shock-transformation-hard/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/ecoli-heat-shock-transformation-hard/tests/spec_check.py +++ b/tasks/ecoli-heat-shock-transformation-hard/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/ecoli-heat-shock-transformation/tests/grade.py b/tasks/ecoli-heat-shock-transformation/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/ecoli-heat-shock-transformation/tests/grade.py +++ b/tasks/ecoli-heat-shock-transformation/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/ecoli-heat-shock-transformation/tests/protocol_lint.py b/tasks/ecoli-heat-shock-transformation/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/ecoli-heat-shock-transformation/tests/protocol_lint.py +++ b/tasks/ecoli-heat-shock-transformation/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/ecoli-heat-shock-transformation/tests/spec_check.py b/tasks/ecoli-heat-shock-transformation/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/ecoli-heat-shock-transformation/tests/spec_check.py +++ b/tasks/ecoli-heat-shock-transformation/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/golden-gate-assembly-hard/tests/grade.py b/tasks/golden-gate-assembly-hard/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/golden-gate-assembly-hard/tests/grade.py +++ b/tasks/golden-gate-assembly-hard/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/golden-gate-assembly-hard/tests/protocol_lint.py b/tasks/golden-gate-assembly-hard/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/golden-gate-assembly-hard/tests/protocol_lint.py +++ b/tasks/golden-gate-assembly-hard/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/golden-gate-assembly-hard/tests/spec_check.py b/tasks/golden-gate-assembly-hard/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/golden-gate-assembly-hard/tests/spec_check.py +++ b/tasks/golden-gate-assembly-hard/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/golden-gate-assembly/tests/grade.py b/tasks/golden-gate-assembly/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/golden-gate-assembly/tests/grade.py +++ b/tasks/golden-gate-assembly/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/golden-gate-assembly/tests/protocol_lint.py b/tasks/golden-gate-assembly/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/golden-gate-assembly/tests/protocol_lint.py +++ b/tasks/golden-gate-assembly/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/golden-gate-assembly/tests/spec_check.py b/tasks/golden-gate-assembly/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/golden-gate-assembly/tests/spec_check.py +++ b/tasks/golden-gate-assembly/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tasks/split-200ul-two-wells/tests/grade.py b/tasks/split-200ul-two-wells/tests/grade.py index f808f68..e90f80f 100644 --- a/tasks/split-200ul-two-wells/tests/grade.py +++ b/tasks/split-200ul-two-wells/tests/grade.py @@ -59,7 +59,7 @@ def grade(protocol: Path = PROTOCOL) -> tuple[dict, dict]: deck = json.loads((TESTS / "deck.json").read_text()) checks_cfg = json.loads((TESTS / "checks.json").read_text()) free = frozenset(checks_cfg["free_wells"]) - res = check(ir, sim, free, deck, frozenset(checks_cfg.get("not_from_paper", []))) + res = check(ir, sim, free, deck, checks_cfg.get("not_from_paper", {})) record["not_applicable"] = res["not_applicable"] passed = sum(c["pass"] for c in res["checks"]) rewards["checks_frac"] = round(passed / len(res["checks"]), 4) diff --git a/tasks/split-200ul-two-wells/tests/protocol_lint.py b/tasks/split-200ul-two-wells/tests/protocol_lint.py index 16ff37d..acbd3f6 100644 --- a/tasks/split-200ul-two-wells/tests/protocol_lint.py +++ b/tasks/split-200ul-two-wells/tests/protocol_lint.py @@ -18,12 +18,20 @@ FLOW_RATE_FIELDS = {"aspirate", "dispense", "blow_out"} +# Settable properties of a pipette (InstrumentContext) in the Opentrons API docs: ordinary protocol code, not patching. +PIPETTE_SETTINGS = {"tip_racks", "starting_tip", "default_speed"} +CLEARANCE_FIELDS = {"aspirate", "dispense"} def _is_flow_rate(t: ast.Attribute) -> bool: - """pipette.flow_rate.aspirate/dispense/blow_out = ... is ordinary Opentrons API use (pipetting speed).""" - return (t.attr in FLOW_RATE_FIELDS and isinstance(t.value, ast.Attribute) and t.value.attr == "flow_rate" - and isinstance(t.value.value, ast.Name)) + """A documented pipette setting on a local variable: p.flow_rate.aspirate/dispense/blow_out, + p.well_bottom_clearance.aspirate/dispense, p.tip_racks, p.starting_tip, p.default_speed. Assigning to anything else, + or to these on a module or class (protocol_api.InstrumentContext.tip_racks = ...), is still refused.""" + if isinstance(t.value, ast.Name): + return t.attr in PIPETTE_SETTINGS + return (isinstance(t.value, ast.Attribute) and isinstance(t.value.value, ast.Name) + and ((t.value.attr == "flow_rate" and t.attr in FLOW_RATE_FIELDS) + or (t.value.attr == "well_bottom_clearance" and t.attr in CLEARANCE_FIELDS))) def violations(source: str) -> list[str]: diff --git a/tasks/split-200ul-two-wells/tests/spec_check.py b/tasks/split-200ul-two-wells/tests/spec_check.py index 7a1e805..4e256e3 100644 --- a/tasks/split-200ul-two-wells/tests/spec_check.py +++ b/tasks/split-200ul-two-wells/tests/spec_check.py @@ -88,6 +88,7 @@ class State: wells: dict = field(default_factory=dict) # (labware, well) -> uL errors: list = field(default_factory=list) # (event index, code) aspirations: int = 0 + fed: dict = field(default_factory=dict) # (labware, well) -> labware it received liquid from def wells_of(event, loadnames=None) -> list[str]: @@ -101,14 +102,15 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f """Apply atomic events in order. `initial` = {(labware, well): uL} for non-stock wells; `ample` = (labware, well) pairs the IR declares as stocks with unlimited volume (aspirating from them is never an error and is not counted).""" st = State(wells=dict(initial or {})) + drawn = set() # labware the current tip has aspirated from for i, e in enumerate(events): kind = e["kind"] if kind == "pick": if st.tip: st.errors.append((i, "already_has_tip")) - st.tip, st.held = True, 0.0 + st.tip, st.held, drawn = True, 0.0, set() elif kind == "drop": - st.tip, st.held = False, 0.0 + st.tip, st.held, drawn = False, 0.0, set() elif kind in ("aspirate", "dispense"): vol = e["volume"] if not st.tip: @@ -125,20 +127,24 @@ def replay(events: list[dict], initial: dict | None = None, ample: frozenset = f break st.wells[(e["labware"], w)] -= vol st.held += vol + drawn.add(e["labware"]) else: if vol > st.held + 1e-9: st.errors.append((i, "overdispense")) for w in wells_of(e, loadnames): st.wells[(e["labware"], w)] = st.wells.get((e["labware"], w), 0.0) + vol + st.fed.setdefault((e["labware"], w), set()).update(drawn - {e["labware"]}) st.held = max(st.held - vol, 0.0) return st def cross_contamination(events: list[dict]) -> list: """(event index, well) where one tip carries liquid between wells it should not. A tip may go back to wells it - dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a well (drawn - from a well it dispensed into), it carries that well's contents: dispensing into any other well, or drawing from - any other well (the stock included), is contamination until a fresh tip.""" + dispensed into (mixing) and to the one well it started from (multi-dispense). Once it has mixed in a destination + (drawn from a well it dispensed into that was not one of its sources), it carries that well's contents: dispensing + into any other well, or drawing from any other well (the stock included), is contamination until a fresh tip. + Mixing in a source before drawing from it (resuspending a colony or beads) carries nothing the transfer is not + meant to move.""" bad, sources, own, carried = [], set(), set(), set() for i, e in enumerate(events): if e["kind"] in ("pick", "drop"): @@ -152,9 +158,9 @@ def cross_contamination(events: list[dict]) -> list: key = (e["labware"], e["well"]) if (sources and key not in sources and key not in own) or carried - {key}: bad.append((i, key[1])) - if key in own: + if key in own and key not in sources: # mixing in a destination carried.add(key) - else: + elif key not in own: sources.add(key) return bad @@ -216,9 +222,11 @@ def _deck_mapping(proto, run, deck, add): def check(proto: Protocol, run: dict, free_wells: frozenset = frozenset(), deck: dict | None = None, - not_from_paper: frozenset = frozenset()) -> dict: + not_from_paper: dict | frozenset = frozenset()) -> dict: """Verdict for one simulated run against an IR. `run` is simulate()'s result; `deck` pins every container to a labware.""" checks = [] + if not isinstance(not_from_paper, dict): + not_from_paper = {c: None for c in not_from_paper} def add(name, ok, detail=""): checks.append({"name": name, "pass": bool(ok), "detail": detail}) @@ -262,12 +270,15 @@ def add(name, ok, detail=""): for cname, lab in mapping.items(): if kinds[cname] in ("reservoir", "waste"): continue - if cname in not_from_paper: # paper-only task whose paper does not fix this container's volumes: rubric only - na.append(f"end_state:{cname} (the source paper does not determine these volumes; judged by the rubric)") - continue want = {(tube_well[cname] if w == "" and tube_well.get(cname) else w): v for (cn, w), v in final.items() if cn == cname and v} only = tube_well.get(cname) # tubes share a rack: look at this tube's well only got = {w: round(v, 6) for (lb, w), v in st.wells.items() if lb == lab and v > 1e-9 and (lb, w) not in ample and (only is None or w == only)} + if cname in not_from_paper: # the paper does not fix these volumes: judge what it does fix + add(f"composition:{cname}", *_composition(cname, lab, want, got, st.fed, proto, mapping, + not_from_paper[cname], + cname in free_wells)) + na.append(f"end_state:{cname} (the source paper does not determine the exact volumes; composition checked instead)") + continue if cname in free_wells: # the instruction does not name the wells: judge the volumes, not the positions ok = sorted(round(v, 6) for v in want.values()) == sorted(got.values()) add(f"end_state:{cname}", ok, f"expected volumes {sorted(want.values())}, got {sorted(got.values())} (positions free)") @@ -280,12 +291,35 @@ def add(name, ok, detail=""): "aspirations": st.aspirations} -def not_from_paper(task_dir) -> frozenset: - """Containers whose end state the task's source paper does not determine, from task.toml [checks].""" +def _composition(cname, lab, want, got, fed, proto, mapping, volume_range, free) -> tuple[bool, str]: + """For a container whose exact volumes the paper does not fix: the same wells are filled (any wells if positions + are free), each from every source container the IR sends there, the wells the IR fills equally hold equal volumes, + and each volume is inside the range the paper supports.""" + problems = [] + if free and len(got) != len(want) or not free and set(got) != set(want): + problems.append(f"filled {len(got)} wells, expected {len(want)}" + ("" if free else f" ({sorted(set(want) ^ set(got))[:4]} differ)")) + need = {mapping[s.source] for s in proto.steps if s.kind == "transfer" and s.dest == cname and s.source in mapping} + short = sorted(w for w in got if need - fed.get((lab, w), set())) + if short: + problems.append(f"{len(short)} wells lack an input, e.g. {short[0]} has none from {sorted(need - fed.get((lab, short[0]), set()))}") + if len(set(want.values())) == 1 and len(set(got.values())) > 1: + problems.append(f"unequal volumes {sorted(set(got.values()))[:4]}") + if volume_range: + lo, hi = volume_range + off = sorted(v for v in set(got.values()) if not lo - 1e-6 <= v <= hi + 1e-6) + if off: + problems.append(f"volumes {off[:3]} outside the {lo:g}-{hi:g} uL the paper supports") + return not problems, "; ".join(problems) + + +def not_from_paper(task_dir) -> dict: + """Containers whose exact end state the task's source paper does not determine, from task.toml [checks]: + {container: (min uL, max uL) the paper supports, or None}. A list of names (no range) is accepted too.""" import tomllib f = pathlib.Path(task_dir) / "task.toml" - return frozenset(tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", [])) if f.exists() else frozenset() + nfp = tomllib.loads(f.read_text()).get("checks", {}).get("not_from_paper", []) if f.exists() else [] + return {c: None for c in nfp} if isinstance(nfp, list) else {c: tuple(r) if r else None for c, r in nfp.items()} def free_wells(task_dir) -> frozenset: diff --git a/tests/test_benchmark_regressions.py b/tests/test_benchmark_regressions.py index b0b0ac1..f6f4e9b 100644 --- a/tests/test_benchmark_regressions.py +++ b/tests/test_benchmark_regressions.py @@ -132,3 +132,32 @@ def test_easy_rna_rubric_holds_the_80ul_the_brief_states(): rubric = json.loads((ROOT / "tasks/opentrons-rna-extraction/tests/rubric.json").read_text()) text = next(i["text"] for i in rubric["task"] if i["id"] == "elution_recovery") assert "Recovering the whole 100 uL fails this item" in text and "Transfer 80 uL of eluate" in text + + +def lint(): + spec = importlib.util.spec_from_file_location("protocol_lint", ROOT / "eval/protocol_lint.py") + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod.violations + + +def test_documented_pipette_settings_pass_the_lint_gate(): + # DeepSeek V4 Pro's a1-a12-100ul protocol set p20.tip_racks = [...] and scored 0 for it (documented Opentrons API) + ok = """metadata = {'apiLevel': '2.15'} +def run(protocol): + p300 = protocol.load_instrument('p300_single_gen2', 'right', tip_racks=[]) + p300.tip_racks = [protocol.load_labware('opentrons_96_tiprack_300ul', 11)] + p300.starting_tip = p300.tip_racks[0]['A1'] + p300.default_speed = 300 + p300.flow_rate.aspirate = 50 + p300.well_bottom_clearance.dispense = 2 +""" + assert lint()(ok) == [] + + +def test_patching_the_api_is_still_refused(): + for bad in ["from opentrons import protocol_api\nprotocol_api.InstrumentContext.tip_racks = None\n", + "from opentrons import protocol_api\nprotocol_api.ProtocolContext.comment = None\n", + "def run(protocol):\n protocol.comment = print\n", + "def run(protocol):\n p = protocol.load_instrument('p300_single_gen2', 'right')\n p.aspirate = None\n"]: + assert any("assignment to an attribute" in v for v in lint()(bad)), bad diff --git a/tests/test_hard_tasks.py b/tests/test_hard_tasks.py index 4ec6623..3aea0d5 100644 --- a/tests/test_hard_tasks.py +++ b/tests/test_hard_tasks.py @@ -73,7 +73,9 @@ def test_every_graded_volume_is_in_the_paper_or_brief_or_derived_from_them(task) def test_colony_hard_does_not_grade_the_reaction_volume_the_paper_does_not_fix(): # ground truth 18 uL master mix + 1 + 1 = 20 uL; the paper's OT-2 reaction is 9 uL mix + 1 uL colony - assert "pcr_plate" in json.loads((ROOT / "tasks/colony-pcr-screening-hard/tests/checks.json").read_text())["not_from_paper"] + nfp = json.loads((ROOT / "tasks/colony-pcr-screening-hard/tests/checks.json").read_text())["not_from_paper"] + quoted("colony-pcr-screening-hard", "9 μL of this master mix") + assert nfp["pcr_plate"] == [9 + 1, 25] # from the paper's OT-2 reaction up to a standard 25 uL Q5 reaction assert "not_from_paper" not in json.loads((ROOT / "tasks/colony-pcr-screening/tests/checks.json").read_text()) diff --git a/tests/test_spec_check.py b/tests/test_spec_check.py index 40fe5b3..ca1aaf8 100644 --- a/tests/test_spec_check.py +++ b/tests/test_spec_check.py @@ -106,3 +106,43 @@ def test_a_tip_that_mixed_in_one_sample_cannot_go_on_to_the_next(): mix_then_next = [pick, asp("A1", "res"), disp("A1"), asp("A1"), disp("A1"), asp("A1", "res"), disp("A2"), drop] assert cross_contamination(multi_dispense) == [] assert cross_contamination(mix_then_next) != [] + + +def test_mixing_a_source_before_drawing_from_it_is_not_contamination(): + """DeepSeek V4 Pro on colony-PCR-hard: fresh tip per colony, mix the colony well, take 1 uL, dispense into its PCR + well. The mix-tracking rule had flagged every colony; only mixing in a destination carries anything.""" + from spec_check import cross_contamination + pick, drop = {"kind": "pick"}, {"kind": "drop"} + asp = lambda w, lw: {"kind": "aspirate", "labware": lw, "well": w} + disp = lambda w, lw: {"kind": "dispense", "labware": lw, "well": w} + colony = [pick, asp("A1", "colonies"), disp("A1", "colonies"), asp("A1", "colonies"), disp("A1", "colonies"), + asp("A1", "colonies"), disp("A1", "pcr"), drop] + assert cross_contamination(colony) == [] + # still caught: mixing in the destination, then back to the stock and on to the next sample + dest_mix_then_stock = [pick, asp("A1", "beads"), disp("A1", "samples"), asp("A1", "samples"), disp("A1", "samples"), + asp("A1", "beads"), disp("B1", "samples"), drop] + assert cross_contamination(dest_mix_then_stock) != [] + + +def test_a_container_the_paper_does_not_fix_is_still_checked_for_what_it_does_fix(): + # colony-PCR-hard: any equal reaction of 10-25 uL passes; a missing input, unequal wells or 40 uL do not + from spec_check import _composition + proto = Protocol.model_validate_json((ROOT / "tasks/colony-pcr-screening-hard/tests/ir.json").read_text()) + mapping = {"master_mix_reservoir": "mm", "colony_plate": "col", "primer_plate": "pri", "pcr_plate": "pcr"} + wells = [r + str(c) for r in "ABCDEFGH" for c in range(1, 13)] + want = {w: 20.0 for w in wells} + fed_all = {("pcr", w): {"mm", "col", "pri"} for w in wells} + ok = lambda got, fed=fed_all: _composition("pcr_plate", "pcr", want, got, fed, proto, mapping, (10, 25), False) + assert ok({w: 11.0 for w in wells})[0] # 9 mix + 1 primer + 1 colony + assert not ok({w: 40.0 for w in wells})[0] + assert not ok({**{w: 11.0 for w in wells}, "H12": 41.0})[0] + assert not ok({w: 11.0 for w in wells[:-1]})[0] + assert not ok({w: 10.0 for w in wells}, {**fed_all, ("pcr", "A1"): {"mm", "col"}})[0] + + +def test_non_claude_cost_comes_from_tokens_at_openrouter_prices(): + sys.path.insert(0, str(ROOT / "scripts")) + from export_run import openrouter_cost + agent = {"n_input_tokens": 1_000_000, "n_cache_tokens": 800_000, "n_output_tokens": 10_000} + pricing = {"prompt": "0.000002", "completion": "0.000006", "input_cache_read": "0.00000025"} + assert openrouter_cost(agent, pricing) == pytest.approx(0.4 + 0.2 + 0.06)