diff --git a/hf_dataset_card.md b/hf_dataset_card.md index d67f1c0..a69a31b 100644 --- a/hf_dataset_card.md +++ b/hf_dataset_card.md @@ -4,7 +4,6 @@ language: license: mit task_categories: - text-generation -- translation tags: - biology - lab-automation @@ -14,7 +13,7 @@ tags: - benchmark - wet-lab - protocol-generation -- claude +- agents pretty_name: Text2WetLab size_categories: - n<1K @@ -22,104 +21,67 @@ size_categories: # Text2WetLab -**Natural language → Opentrons OT-2 robot protocol benchmark** +**Benchmarking LLM agents on turning lab protocols and papers into Opentrons OT-2 robot code.** -Text2WetLab evaluates whether large language model agents can translate plain-English wet-lab procedure descriptions into correct, safe, executable liquid-handling protocols for the Opentrons OT-2. +Code that passes the vendor simulator can still ruin an experiment: it can pipette-mix fragile cells, carry beads into +an eluate, or draw more from a reservoir than it holds. Text2WetLab grades whether an agent's protocol does what the +protocol or paper says. It has 11 tasks in the [Harbor](https://github.com/harbor-framework/harbor) format at two levels: +*easy* tasks give the exact steps; *hard* tasks give only a goal, a fixed deck and the source paper. -[![GitHub](https://img.shields.io/badge/GitHub-PhysicalAIBenchmarks%2FText2WetLab-black)](https://github.com/PhysicalAIBenchmarks/Text2WetLab) -[![Leaderboard](https://img.shields.io/badge/Leaderboard-Live-teal)](https://physicalaibenchmarks.github.io/Text2WetLab/leaderboard.html) +[Preprint](https://physicalaibenchmarks.github.io/Text2WetLab/docs/preprint/preprint.html) · +[Leaderboard](https://physicalaibenchmarks.github.io/Text2WetLab/leaderboard.html) · +[GitHub](https://github.com/PhysicalAIBenchmarks/Text2WetLab) · +[All links](https://github.com/PhysicalAIBenchmarks/Text2WetLab/blob/main/docs/urls.md) -## Reference episode — MuJoCo 3D physics render +![Trailer preview](assets/trailer_preview.gif) -![OT-2 trailer preview](assets/trailer_preview.gif) +## Results -## Results (pass@1, 2026-10-04) + -3 Claude models evaluated as Claude Code agents in Harbor sandboxes on Modal. All 21 trials passed the Opentrons simulator gate; all deterministic end-state checks passed. Score variation comes entirely from the per-task LLM rubric judge. +Refused tasks are not scored: a refusal is the provider's safety policy, not a protocol. Models are ranked on the +tasks every model answered; Opus 5.5 and Fable 5.1 refused some standard cloning prompts (Anthropic's biosecurity +filter), and those are reported, not scored. No model failed the simulator or a reward-hacking trap; the points are +lost to the rubric, mostly for inventing steps or quantities that the paper does not support. -| Model | Mean reward | Cost (7 tasks) | -|---|---|---| -| **claude-opus-5-5** | **0.935** | $1.25 | -| claude-sonnet-5-5 | 0.910 | $0.41 | -| claude-fable-5-1 | 0.882 | $3.69 | +## What is in this dataset -### Per-task breakdown +What an agent is given, for all 11 tasks: -| Task | Opus 5.5 | Sonnet 5.5 | Fable 5.1 | Steps | Source | -|---|---|---|---|---|---| -| `a1-a12-100ul` | 1.000 | 0.917 | 0.833 | 1 | handwritten | -| `split-200ul-two-wells` | 1.000 | 1.000 | 0.917 | 1 | handwritten | -| `ampure-bead-cleanup` | 0.944 | 0.889 | **1.000** | 13 | pioneer-ngs | -| `colony-pcr-screening` | **1.000** | 0.875 | 0.875 | 4 | slowpoke/apex | -| `ecoli-heat-shock-transformation` | 0.714 | **0.857** | 0.714 | 4 | apex | -| `golden-gate-assembly` | **1.000** | **1.000** | **1.000** | 33 | assemblytron | -| `opentrons-rna-extraction` | **0.889** | 0.833 | 0.833 | 48-sample | hulp-rna-extraction | +- `tasks//instruction.md`: the brief: fixed deck, starting contents, and the steps (easy) or a goal and the + source paper (hard). +- `tasks//task.toml`: Harbor manifest, source papers, metadata. +- `tasks//public/`: for the easy tasks built from a protocol IR, the IR (`ir.json`) and what the instruction + leaves out (`assumptions.md`). +- `sources/`: one record per source paper (48), with `master.csv`; papers are linked by DOI and checked by SHA-256, + never redistributed. +- `eval/`, `assets/`, `docs/`, `manuscript/`: simulator and renderer code, visualisations, documentation, the preprint + source. -### Key findings - -- **100% simulator pass rate** — all 21 trials passed `opentrons_simulate`; structural correctness is solved for frontier models -- **`robot_practice` is the universal weak point** — penalised in 11/21 trials (52%); encodes tacit wet-lab knowledge not explicit in task specs -- **Phantom mix hallucination** — Opus 5.5 and Fable 5.1 added unrequested `mix_after` to competent cells in the E. coli transformation task (biologically harmful) -- **Dispensing practice only the judge sees**: in colony PCR, every trial passed `no_cross_contamination`. Sonnet 5.5 and Fable 5.1 lost points for running all 96 master-mix dispenses from one tip without contact control, and for default heights on 1 µL additions. -- **The reproducibility gap**: in RNA extraction, all models recovered 100 µL of eluate. The paper says only "collect the supernatant", while the authors' deposited OT-2 script recovers 80 µL. An agent that reads only the paper cannot recover that parameter. -- **Sonnet 5.5 = best cost-efficiency** — 97.3% of Opus 5.5 performance at 33% cost +Not in this dataset: the graders, ground truth checks, rubrics and reference solutions (`tests/`, `solution/`), the task +containers (`environment/`), third-party author code, and paper PDFs. This copy mirrors that allowlist on every deploy +from `main`. To run the benchmark, clone the GitHub repository and use Harbor. ## Grading -Each task uses a two-layer grader: - -1. **Deterministic gate**: `protocol_lint.py` → `opentrons_simulate` → `spec_check.py` (exact labware end-state volume checks) -2. **LLM rubric judge**: `claude-sonnet-5-5` scores 5–9 task-specific rubric items (0 / 0.5 / 1) - -Final reward = (deterministic_reward + judge_mean) / 2 - -Rubric items are task-specific (e.g. `dna_addition`, `heat_shock`, `soc_recovery` for E. coli transformation; `binding`, `washes`, `drying_and_elution` for AMPure cleanup). All tasks share `deck_and_hardware` and `robot_practice`. - -## Tasks - -One folder per task under `tasks//`: -- `public/instruction.md` — what the model is given -- `public/ir.json` — the decomposed Protocol IR spec -- `public/assumptions.md` — what the instruction leaves out -- `task.toml` — manifest, source paper, Harbor schema - -Oracle protocols and graders are not published (to prevent contamination of future evaluations). - -## 2D IR visualisations - -Step-by-step liquid tracking from the Protocol IR: - -| `a1-a12-100ul` | `ampure-bead-cleanup` | `colony-pcr-screening` | -|---|---|---| -| ![](assets/examples/a1-a12-100ul.gif) | ![](assets/examples/ampure-bead-cleanup.gif) | ![](assets/examples/colony-pcr-screening.gif) | - -| `ecoli-heat-shock-transformation` | `golden-gate-assembly` | `split-200ul-two-wells` | -|---|---|---| -| ![](assets/examples/ecoli-heat-shock-transformation.gif) | ![](assets/examples/golden-gate-assembly.gif) | ![](assets/examples/split-200ul-two-wells.gif) | - -## 3D MuJoCo renders - -Oracle protocol execution in the opentrons-mujoco-viz physics engine: - -| `a1-a12-100ul` | `golden-gate-assembly` | `ecoli-heat-shock-transformation` | -|---|---|---| -| ![](assets/examples3d/a1-a12-100ul.gif) | ![](assets/examples3d/golden-gate-assembly.gif) | ![](assets/examples3d/ecoli-heat-shock-transformation.gif) | - -## Ingestion pipeline - -Papers are ingested via `paper2protocol`: PDF/DOI → LLM extraction → Protocol IR → `opentrons_simulate` validation → task. The same pipeline can reproduce any task from its source paper. +A layered verifier: a lint gate, 10 reward-hacking traps, the Opentrons 7.5 simulator, deterministic checks of the +simulated run against the ground truth (labware, every end-state volume, tip use, contamination, reservoir capacity), +then a pass/fail rubric scored by an LLM judge as the majority of three calls. Three core items (robot practice, tips +and contamination, fidelity to the task or paper) carry 75% of the reward; task items share the rest. A failed critical +check caps the reward at 0.3. All 11 reference solutions pass; 152 broken or cheating protocols all fail. ## Citation ```bibtex -@misc{oleary2026text2wetlab, - title = {Text2WetLab: Benchmarking Large Language Model Agents on Opentrons OT-2 Protocol Generation}, +@misc{text2wetlab2026, + title = {Text2WetLab: Benchmarking LLM Agents on Turning Lab Protocols and Papers into Robot Code}, author = {O'Leary, Evan and Alshehri, Mohammed and Legon, Laurence}, year = {2026}, url = {https://github.com/PhysicalAIBenchmarks/Text2WetLab} } ``` -## License +## Licence -Apache 2.0. Source paper PDFs are not redistributed; they are fetched by DOI and verified by SHA-256. See `PROVENANCE.csv`. +MIT for the code and task files. Paper text and third-party code keep their own licences and are not redistributed +here. diff --git a/scripts/deploy_hf.py b/scripts/deploy_hf.py index 954b577..4172dec 100644 --- a/scripts/deploy_hf.py +++ b/scripts/deploy_hf.py @@ -1,6 +1,10 @@ """Upload the public dataset to the HuggingFace Hub. Reads HF_TOKEN from the environment. - python scripts/deploy_hf.py --dry-run # stage and list what would be published + python scripts/deploy_hf.py --dry-run # stage, list what would be published and what would be deleted + +The Hub copy mirrors the allowlist: files on the Hub that are no longer published (an old layout, a path the allowlist +dropped) are deleted in the same deploy. The dataset card is hf_dataset_card.md with its results table generated from +the latest results/runs/ folder, the same data as the leaderboard. Publishing is an allowlist. Never published: - tasks//private/ and harbor/ hidden grader, oracle and sandbox files (contamination) @@ -9,6 +13,7 @@ - data/, .git, caches generated or internal """ import argparse +import importlib.util import os import shutil import tempfile @@ -20,6 +25,8 @@ HIDDEN = {"private", "harbor", "tests", "solution", "environment"} # hidden inside tasks// NOISE = {"__pycache__", ".DS_Store"} NEVER_PUBLISHED_SUFFIXES = (".pdf",) # papers are fetched by URL and checked by SHA-256; licences differ per paper +KEEP_ON_HUB = {".gitattributes"} # created by the Hub; never ours to delete +RESULTS_MARKER = "" def _ignore(directory, names): @@ -32,6 +39,35 @@ def _ignore(directory, names): return skip +def results_markdown() -> str: + """The latest run's summary, from the same function that builds the leaderboard.""" + spec = importlib.util.spec_from_file_location("build_site", ROOT / "scripts/build_site.py") + site = importlib.util.module_from_spec(spec) + spec.loader.exec_module(site) + run = sorted((ROOT / "results/runs").iterdir())[-1] + d = site.run_data(run) + fmt = lambda v: "-" if v is None else f"{v:.3f}" + lines = [f"Run `{d['run']}`: {len(d['tasks'])} tasks, one attempt each, the same Claude Code agent for every model, " + "Claude Sonnet 5.5 as a three-vote judge. Full results: " + "[leaderboard](https://physicalaibenchmarks.github.io/Text2WetLab/leaderboard.html), " + f"[report](https://github.com/PhysicalAIBenchmarks/Text2WetLab/blob/main/results/runs/{d['run']}/REPORT.md).", + "", + f"| Model | **Mean, {d['n_common']} tasks all answered** | Mean, all tasks answered | Easy | Paper-only | Refused, not scored | Agent cost |", + "|---|:---:|:---:|:---:|:---:|:---:|:---:|"] + for r in sorted(d["rows"], key=lambda r: -(r["common"] or 0)): # refused tasks are not scored + lines.append(f"| {r['name']} | **{fmt(r['common'])}** | {fmt(r['answered'])} ({r['n_answered']}) | " + f"{fmt(r['easy'])} | {fmt(r['hard'])} | {r['refusals']} | ${r['cost']:.2f} |") + if d.get("n_provisional"): + lines += ["", f"{d['n_provisional']} of these scores are provisional until a re-judge with the audited verifier " + f"(reasons in `results/runs/{d['run']}/PROVISIONAL.json`; marked † on the leaderboard)."] + return "\n".join(lines) + + +def stale_files(on_hub: set[str], published: set[str]) -> list[str]: + """Files on the Hub that this deploy does not publish: deleted so the Hub mirrors the allowlist.""" + return sorted(on_hub - published - KEEP_ON_HUB) + + def stage(dest: Path) -> list[str]: """Copy the public files into dest; return their relative paths.""" for name in INCLUDE: @@ -40,7 +76,8 @@ def stage(dest: Path) -> list[str]: shutil.copytree(src, dest / name, ignore=_ignore) elif src.exists(): shutil.copy(src, dest / name) - shutil.copy(ROOT / "hf_dataset_card.md", dest / "README.md") + card = (ROOT / "hf_dataset_card.md").read_text() + (dest / "README.md").write_text(card.replace(RESULTS_MARKER, results_markdown())) return sorted(str(p.relative_to(dest)) for p in dest.rglob("*") if p.is_file()) @@ -50,16 +87,25 @@ def main(): args = ap.parse_args() with tempfile.TemporaryDirectory() as tmp: files = stage(Path(tmp)) + from huggingface_hub import CommitOperationDelete, HfApi + + api = HfApi(token=os.environ.get("HF_TOKEN")) + try: + on_hub = set(api.list_repo_files(REPO_ID, repo_type="dataset")) + except Exception: # first deploy, or no access in a dry run + on_hub = set() + stale = stale_files(on_hub, set(files)) if args.dry_run: print("\n".join(files)) - print(f"{len(files)} files would be published to {REPO_ID}") + print("\n".join(f"DELETE {f}" for f in stale)) + print(f"{len(files)} files would be published to {REPO_ID}, {len(stale)} deleted") return - from huggingface_hub import HfApi - - api = HfApi(token=os.environ["HF_TOKEN"]) api.create_repo(REPO_ID, repo_type="dataset", exist_ok=True) api.upload_folder(folder_path=tmp, repo_id=REPO_ID, repo_type="dataset", commit_message="Deploy from GitHub") - print(f"Deployed to https://huggingface.co/datasets/{REPO_ID}") + if stale: + api.create_commit(REPO_ID, repo_type="dataset", operations=[CommitOperationDelete(path_in_repo=f) for f in stale], + commit_message=f"Remove {len(stale)} files no longer published") + print(f"Deployed to https://huggingface.co/datasets/{REPO_ID}: {len(files)} files, {len(stale)} removed") if __name__ == "__main__": diff --git a/tests/test_deploy.py b/tests/test_deploy.py index 5cec77e..5d2d92b 100644 --- a/tests/test_deploy.py +++ b/tests/test_deploy.py @@ -33,3 +33,26 @@ def test_master_csv_is_published_and_no_pdf_ever_is(tmp_path): files = staged(tmp_path) assert "sources/master.csv" in files and "sources/sources.json" in files assert not [f for f in files if f.lower().endswith(".pdf")] + + +def test_the_hub_copy_mirrors_the_allowlist(): + # an old layout's grader files and third-party scripts stayed on the Hub because uploads never deleted + on_hub = {".gitattributes", "README.md", "tasks/harbor/a1-a12-100ul/tests/reference_protocol.py", + "ref/dna-bot-ysaa010/scripts/1_clip.ot2.py", "tasks/a1-a12-100ul/task.toml"} + published = {"README.md", "tasks/a1-a12-100ul/task.toml"} + assert deploy.stale_files(on_hub, published) == ["ref/dna-bot-ysaa010/scripts/1_clip.ot2.py", + "tasks/harbor/a1-a12-100ul/tests/reference_protocol.py"] + + +def test_no_grader_or_solution_file_is_staged_for_any_task(tmp_path): + files = staged(tmp_path) + hidden = [f for f in files if f.startswith("tasks/") and any(f"/{d}/" in f for d in ("tests", "solution", "environment"))] + assert not hidden, hidden + + +def test_the_card_carries_the_latest_results_and_one_licence(tmp_path): + staged(tmp_path) + card = (tmp_path / "README.md").read_text() + assert deploy.RESULTS_MARKER not in card and "| GPT-6.1 Sol |" in card and "tasks all answered" in card + assert "refusals as 0" not in card and "Refused, not scored" in card + assert "license: mit" in card and "Apache" not in card