Tyronita · Oct 7, 2026 · Oct 7, 2026 · Oct 7, 2026 · Oct 7, 2026
diff --git a/hf_dataset_card.md b/hf_dataset_card.md
@@ -4,7 +4,6 @@ language:
 license: mit
 task_categories:
 - text-generation
-- translation
 tags:
 - biology
 - lab-automation
@@ -14,112 +13,75 @@ tags:
 - benchmark
 - wet-lab
 - protocol-generation
-- claude
+- agents
 pretty_name: Text2WetLab
 size_categories:
 - n<1K
 ---
 
 # Text2WetLab
 
-**Natural language → Opentrons OT-2 robot protocol benchmark**
+**Benchmarking LLM agents on turning lab protocols and papers into Opentrons OT-2 robot code.**
 
-Text2WetLab evaluates whether large language model agents can translate plain-English wet-lab procedure descriptions into correct, safe, executable liquid-handling protocols for the Opentrons OT-2.
+Code that passes the vendor simulator can still ruin an experiment: it can pipette-mix fragile cells, carry beads into
+an eluate, or draw more from a reservoir than it holds. Text2WetLab grades whether an agent's protocol does what the
+protocol or paper says. It has 11 tasks in the [Harbor](https://github.com/harbor-framework/harbor) format at two levels:
+*easy* tasks give the exact steps; *hard* tasks give only a goal, a fixed deck and the source paper.
 
-[![GitHub](https://img.shields.io/badge/GitHub-PhysicalAIBenchmarks%2FText2WetLab-black)](https://github.com/PhysicalAIBenchmarks/Text2WetLab)
-[![Leaderboard](https://img.shields.io/badge/Leaderboard-Live-teal)](https://physicalaibenchmarks.github.io/Text2WetLab/leaderboard.html)
+[Preprint](https://physicalaibenchmarks.github.io/Text2WetLab/docs/preprint/preprint.html) ·
+[Leaderboard](https://physicalaibenchmarks.github.io/Text2WetLab/leaderboard.html) ·
+[GitHub](https://github.com/PhysicalAIBenchmarks/Text2WetLab) ·
+[All links](https://github.com/PhysicalAIBenchmarks/Text2WetLab/blob/main/docs/urls.md)
 
-## Reference episode — MuJoCo 3D physics render
+![Trailer preview](assets/trailer_preview.gif)
 
-![OT-2 trailer preview](assets/trailer_preview.gif)
+## Results
 
-## Results (pass@1, 2026-10-04)
+<!-- RESULTS -->
 
-3 Claude models evaluated as Claude Code agents in Harbor sandboxes on Modal. All 21 trials passed the Opentrons simulator gate; all deterministic end-state checks passed. Score variation comes entirely from the per-task LLM rubric judge.
+Refused tasks are not scored: a refusal is the provider's safety policy, not a protocol. Models are ranked on the
+tasks every model answered; Opus 5.5 and Fable 5.1 refused some standard cloning prompts (Anthropic's biosecurity
+filter), and those are reported, not scored. No model failed the simulator or a reward-hacking trap; the points are
+lost to the rubric, mostly for inventing steps or quantities that the paper does not support.
 
-| Model | Mean reward | Cost (7 tasks) |
-|---|---|---|
-| **claude-opus-5-5** | **0.935** | $1.25 |
-| claude-sonnet-5-5 | 0.910 | $0.41 |
-| claude-fable-5-1 | 0.882 | $3.69 |
+## What is in this dataset
 
-### Per-task breakdown
+What an agent is given, for all 11 tasks:
 
-| Task | Opus 5.5 | Sonnet 5.5 | Fable 5.1 | Steps | Source |
-|---|---|---|---|---|---|
-| `a1-a12-100ul` | 1.000 | 0.917 | 0.833 | 1 | handwritten |
-| `split-200ul-two-wells` | 1.000 | 1.000 | 0.917 | 1 | handwritten |
-| `ampure-bead-cleanup` | 0.944 | 0.889 | **1.000** | 13 | pioneer-ngs |
-| `colony-pcr-screening` | **1.000** | 0.875 | 0.875 | 4 | slowpoke/apex |
-| `ecoli-heat-shock-transformation` | 0.714 | **0.857** | 0.714 | 4 | apex |
-| `golden-gate-assembly` | **1.000** | **1.000** | **1.000** | 33 | assemblytron |
-| `opentrons-rna-extraction` | **0.889** | 0.833 | 0.833 | 48-sample | hulp-rna-extraction |
+- `tasks/<task>/instruction.md`: the brief: fixed deck, starting contents, and the steps (easy) or a goal and the
+  source paper (hard).
+- `tasks/<task>/task.toml`: Harbor manifest, source papers, metadata.
+- `tasks/<task>/public/`: for the easy tasks built from a protocol IR, the IR (`ir.json`) and what the instruction
+  leaves out (`assumptions.md`).
+- `sources/`: one record per source paper (48), with `master.csv`; papers are linked by DOI and checked by SHA-256,
+  never redistributed.
+- `eval/`, `assets/`, `docs/`, `manuscript/`: simulator and renderer code, visualisations, documentation, the preprint
+  source.
 
-### Key findings
-
-- **100% simulator pass rate** — all 21 trials passed `opentrons_simulate`; structural correctness is solved for frontier models
-- **`robot_practice` is the universal weak point** — penalised in 11/21 trials (52%); encodes tacit wet-lab knowledge not explicit in task specs
-- **Phantom mix hallucination** — Opus 5.5 and Fable 5.1 added unrequested `mix_after` to competent cells in the E. coli transformation task (biologically harmful)
-- **Dispensing practice only the judge sees**: in colony PCR, every trial passed `no_cross_contamination`. Sonnet 5.5 and Fable 5.1 lost points for running all 96 master-mix dispenses from one tip without contact control, and for default heights on 1 µL additions.
-- **The reproducibility gap**: in RNA extraction, all models recovered 100 µL of eluate. The paper says only "collect the supernatant", while the authors' deposited OT-2 script recovers 80 µL. An agent that reads only the paper cannot recover that parameter.
-- **Sonnet 5.5 = best cost-efficiency** — 97.3% of Opus 5.5 performance at 33% cost
+Not in this dataset: the graders, ground truth checks, rubrics and reference solutions (`tests/`, `solution/`), the task
+containers (`environment/`), third-party author code, and paper PDFs. This copy mirrors that allowlist on every deploy
+from `main`. To run the benchmark, clone the GitHub repository and use Harbor.
 
 ## Grading
 
-Each task uses a two-layer grader:
-
-1. **Deterministic gate**: `protocol_lint.py` → `opentrons_simulate` → `spec_check.py` (exact labware end-state volume checks)
-2. **LLM rubric judge**: `claude-sonnet-5-5` scores 5–9 task-specific rubric items (0 / 0.5 / 1)
-
-Final reward = (deterministic_reward + judge_mean) / 2
-
-Rubric items are task-specific (e.g. `dna_addition`, `heat_shock`, `soc_recovery` for E. coli transformation; `binding`, `washes`, `drying_and_elution` for AMPure cleanup). All tasks share `deck_and_hardware` and `robot_practice`.
-
-## Tasks
-
-One folder per task under `tasks/<task>/`:
-- `public/instruction.md` — what the model is given
-- `public/ir.json` — the decomposed Protocol IR spec
-- `public/assumptions.md` — what the instruction leaves out
-- `task.toml` — manifest, source paper, Harbor schema
-
-Oracle protocols and graders are not published (to prevent contamination of future evaluations).
-
-## 2D IR visualisations
-
-Step-by-step liquid tracking from the Protocol IR:
-
-| `a1-a12-100ul` | `ampure-bead-cleanup` | `colony-pcr-screening` |
-|---|---|---|
-| ![](assets/examples/a1-a12-100ul.gif) | ![](assets/examples/ampure-bead-cleanup.gif) | ![](assets/examples/colony-pcr-screening.gif) |
-
-| `ecoli-heat-shock-transformation` | `golden-gate-assembly` | `split-200ul-two-wells` |
-|---|---|---|
-| ![](assets/examples/ecoli-heat-shock-transformation.gif) | ![](assets/examples/golden-gate-assembly.gif) | ![](assets/examples/split-200ul-two-wells.gif) |
-
-## 3D MuJoCo renders
-
-Oracle protocol execution in the opentrons-mujoco-viz physics engine:
-
-| `a1-a12-100ul` | `golden-gate-assembly` | `ecoli-heat-shock-transformation` |
-|---|---|---|
-| ![](assets/examples3d/a1-a12-100ul.gif) | ![](assets/examples3d/golden-gate-assembly.gif) | ![](assets/examples3d/ecoli-heat-shock-transformation.gif) |
-
-## Ingestion pipeline
-
-Papers are ingested via `paper2protocol`: PDF/DOI → LLM extraction → Protocol IR → `opentrons_simulate` validation → task. The same pipeline can reproduce any task from its source paper.
+A layered verifier: a lint gate, 10 reward-hacking traps, the Opentrons 7.5 simulator, deterministic checks of the
+simulated run against the ground truth (labware, every end-state volume, tip use, contamination, reservoir capacity),
+then a pass/fail rubric scored by an LLM judge as the majority of three calls. Three core items (robot practice, tips
+and contamination, fidelity to the task or paper) carry 75% of the reward; task items share the rest. A failed critical
+check caps the reward at 0.3. All 11 reference solutions pass; 152 broken or cheating protocols all fail.
 
 ## Citation
 
 ```bibtex
-@misc{oleary2026text2wetlab,
-  title  = {Text2WetLab: Benchmarking Large Language Model Agents on Opentrons OT-2 Protocol Generation},
+@misc{text2wetlab2026,
+  title  = {Text2WetLab: Benchmarking LLM Agents on Turning Lab Protocols and Papers into Robot Code},
   author = {O'Leary, Evan and Alshehri, Mohammed and Legon, Laurence},
   year   = {2026},
   url    = {https://github.com/PhysicalAIBenchmarks/Text2WetLab}
 }
 ```
 
-## License
+## Licence
 
-Apache 2.0. Source paper PDFs are not redistributed; they are fetched by DOI and verified by SHA-256. See `PROVENANCE.csv`.
+MIT for the code and task files. Paper text and third-party code keep their own licences and are not redistributed
+here.
diff --git a/scripts/deploy_hf.py b/scripts/deploy_hf.py
@@ -1,6 +1,10 @@
 """Upload the public dataset to the HuggingFace Hub. Reads HF_TOKEN from the environment.
 
-    python scripts/deploy_hf.py --dry-run      # stage and list what would be published
+    python scripts/deploy_hf.py --dry-run      # stage, list what would be published and what would be deleted
+
+The Hub copy mirrors the allowlist: files on the Hub that are no longer published (an old layout, a path the allowlist
+dropped) are deleted in the same deploy. The dataset card is hf_dataset_card.md with its results table generated from
+the latest results/runs/ folder, the same data as the leaderboard.
 
 Publishing is an allowlist. Never published:
   - tasks/<task>/private/ and harbor/   hidden grader, oracle and sandbox files (contamination)
@@ -9,6 +13,7 @@
   - data/, .git, caches                 generated or internal
 """
 import argparse
+import importlib.util
 import os
 import shutil
 import tempfile
@@ -20,6 +25,8 @@
 HIDDEN = {"private", "harbor", "tests", "solution", "environment"}  # hidden inside tasks/<task>/
 NOISE = {"__pycache__", ".DS_Store"}
 NEVER_PUBLISHED_SUFFIXES = (".pdf",)  # papers are fetched by URL and checked by SHA-256; licences differ per paper
+KEEP_ON_HUB = {".gitattributes"}       # created by the Hub; never ours to delete
+RESULTS_MARKER = "<!-- RESULTS -->"
 
 
 def _ignore(directory, names):
@@ -32,6 +39,35 @@ def _ignore(directory, names):
     return skip
 
 
+def results_markdown() -> str:
+    """The latest run's summary, from the same function that builds the leaderboard."""
+    spec = importlib.util.spec_from_file_location("build_site", ROOT / "scripts/build_site.py")
+    site = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(site)
+    run = sorted((ROOT / "results/runs").iterdir())[-1]
+    d = site.run_data(run)
+    fmt = lambda v: "-" if v is None else f"{v:.3f}"
+    lines = [f"Run `{d['run']}`: {len(d['tasks'])} tasks, one attempt each, the same Claude Code agent for every model, "
+             "Claude Sonnet 5.5 as a three-vote judge. Full results: "
+             "[leaderboard](https://physicalaibenchmarks.github.io/Text2WetLab/leaderboard.html), "
+             f"[report](https://github.com/PhysicalAIBenchmarks/Text2WetLab/blob/main/results/runs/{d['run']}/REPORT.md).",
+             "",
+             f"| Model | **Mean, {d['n_common']} tasks all answered** | Mean, all tasks answered | Easy | Paper-only | Refused, not scored | Agent cost |",
+             "|---|:---:|:---:|:---:|:---:|:---:|:---:|"]
+    for r in sorted(d["rows"], key=lambda r: -(r["common"] or 0)):   # refused tasks are not scored
+        lines.append(f"| {r['name']} | **{fmt(r['common'])}** | {fmt(r['answered'])} ({r['n_answered']}) | "
+                     f"{fmt(r['easy'])} | {fmt(r['hard'])} | {r['refusals']} | ${r['cost']:.2f} |")
+    if d.get("n_provisional"):
+        lines += ["", f"{d['n_provisional']} of these scores are provisional until a re-judge with the audited verifier "
+                  f"(reasons in `results/runs/{d['run']}/PROVISIONAL.json`; marked † on the leaderboard)."]
+    return "\n".join(lines)
+
+
+def stale_files(on_hub: set[str], published: set[str]) -> list[str]:
+    """Files on the Hub that this deploy does not publish: deleted so the Hub mirrors the allowlist."""
+    return sorted(on_hub - published - KEEP_ON_HUB)
+
+
 def stage(dest: Path) -> list[str]:
     """Copy the public files into dest; return their relative paths."""
     for name in INCLUDE:
@@ -40,7 +76,8 @@ def stage(dest: Path) -> list[str]:
             shutil.copytree(src, dest / name, ignore=_ignore)
         elif src.exists():
             shutil.copy(src, dest / name)
-    shutil.copy(ROOT / "hf_dataset_card.md", dest / "README.md")
+    card = (ROOT / "hf_dataset_card.md").read_text()
+    (dest / "README.md").write_text(card.replace(RESULTS_MARKER, results_markdown()))
     return sorted(str(p.relative_to(dest)) for p in dest.rglob("*") if p.is_file())
 
 
@@ -50,16 +87,25 @@ def main():
     args = ap.parse_args()
     with tempfile.TemporaryDirectory() as tmp:
         files = stage(Path(tmp))
+        from huggingface_hub import CommitOperationDelete, HfApi
+
+        api = HfApi(token=os.environ.get("HF_TOKEN"))
+        try:
+            on_hub = set(api.list_repo_files(REPO_ID, repo_type="dataset"))
+        except Exception:  # first deploy, or no access in a dry run
+            on_hub = set()
+        stale = stale_files(on_hub, set(files))
         if args.dry_run:
             print("\n".join(files))
-            print(f"{len(files)} files would be published to {REPO_ID}")
+            print("\n".join(f"DELETE {f}" for f in stale))
+            print(f"{len(files)} files would be published to {REPO_ID}, {len(stale)} deleted")
             return
-        from huggingface_hub import HfApi
-
-        api = HfApi(token=os.environ["HF_TOKEN"])
         api.create_repo(REPO_ID, repo_type="dataset", exist_ok=True)
         api.upload_folder(folder_path=tmp, repo_id=REPO_ID, repo_type="dataset", commit_message="Deploy from GitHub")
-    print(f"Deployed to https://huggingface.co/datasets/{REPO_ID}")
+        if stale:
+            api.create_commit(REPO_ID, repo_type="dataset", operations=[CommitOperationDelete(path_in_repo=f) for f in stale],
+                              commit_message=f"Remove {len(stale)} files no longer published")
+    print(f"Deployed to https://huggingface.co/datasets/{REPO_ID}: {len(files)} files, {len(stale)} removed")
 
 
 if __name__ == "__main__":

diff --git a/tests/test_deploy.py b/tests/test_deploy.py
@@ -33,3 +33,26 @@ def test_master_csv_is_published_and_no_pdf_ever_is(tmp_path):
     files = staged(tmp_path)
     assert "sources/master.csv" in files and "sources/sources.json" in files
     assert not [f for f in files if f.lower().endswith(".pdf")]
+
+
+def test_the_hub_copy_mirrors_the_allowlist():
+    # an old layout's grader files and third-party scripts stayed on the Hub because uploads never deleted
+    on_hub = {".gitattributes", "README.md", "tasks/harbor/a1-a12-100ul/tests/reference_protocol.py",
+              "ref/dna-bot-ysaa010/scripts/1_clip.ot2.py", "tasks/a1-a12-100ul/task.toml"}
+    published = {"README.md", "tasks/a1-a12-100ul/task.toml"}
+    assert deploy.stale_files(on_hub, published) == ["ref/dna-bot-ysaa010/scripts/1_clip.ot2.py",
+                                                     "tasks/harbor/a1-a12-100ul/tests/reference_protocol.py"]
+
+
+def test_no_grader_or_solution_file_is_staged_for_any_task(tmp_path):
+    files = staged(tmp_path)
+    hidden = [f for f in files if f.startswith("tasks/") and any(f"/{d}/" in f for d in ("tests", "solution", "environment"))]
+    assert not hidden, hidden
+
+
+def test_the_card_carries_the_latest_results_and_one_licence(tmp_path):
+    staged(tmp_path)
+    card = (tmp_path / "README.md").read_text()
+    assert deploy.RESULTS_MARKER not in card and "| GPT-6.1 Sol |" in card and "tasks all answered" in card
+    assert "refusals as 0" not in card and "Refused, not scored" in card
+    assert "license: mit" in card and "Apache" not in card