Repository navigation
Address Laurence's README review (#61); one source for the paper; related-work survey; drift test - #64
Closed
Tyronita wants to merge 6 commits into
Closed
Address Laurence's README review (#61); one source for the paper; related-work survey; drift test#64Tyronita wants to merge 6 commits into
Tyronita wants to merge 6 commits into
Conversation
Reviewing readme file for completeness
The Sonnet, Opus, Fable and GPT records for opentrons-rna-extraction predated the check, so REPORT.md and README Table 7 counted 32 RNA run-log checks for them and 33 for Qwen and DeepSeek (spotted in review of #61). Re-ran the run-log checks on the saved protocols with Opentrons 7.5.0 (Qwen's rerun reproduces its stored record exactly): all four pass recover_about_80ul and every other check is unchanged. No reward changes; Sonnet's deterministic_reward goes from 0.4706 to 0.4722 (17/18 checks, capped). REPORT.md and Figure 8 regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Abstract cut to ~200 words; introduction now goes problem, state of the art (LAB-Bench, BioPlanner, LabscriptAI, ProtoPilot, ABC-Bench, AEGIS), then one paragraph on what Text2WetLab does. Contributions: six models, four providers. - "Paper-only" used throughout for the harder level (task names keep the -hard suffix). - Traps: 9, not 10. "Fake comments" was never a trap (anti_hack.py has 9); the judge fails them under fidelity. Fixed in README, manuscript, site and Figure 1. - Figure 1 redrawn: panels A/B, six models, verifier steps described, no file names; legend readable on its own. Figure 3 shows two labelled frames instead of twelve. - Grader-fix table and the Docker/Modal timing table moved to Appendices D and E; 5.5 is now about run-to-run noise. - Cost also reported on the 8 tasks every model answered (Table 5, Figure 7b). - Refusals no longer called false positives; notes Opus answered the transformation prompt Fable refused. - RNA-hard full 100 uL recovery described as poor OT-2 practice the rubric does not yet penalise. - Simulator pass rate qualified: easy-task instructions tell the agent to run opentrons_simulate. - Fidelity framed as the item that matters most. - Removed the unsupported "gaps under ~0.1 are not meaningful" claim. - New limitations: one harness, prompt help, fixed decks, published code (training leakage, reporting bias), RNA recovery. - Table 6: RNA run-log checks now 33/33 for every model (see previous commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ft test Unify - manuscript/main.md is the only source of the paper. build.sh now also writes manuscript/main.tex (natbib + refs.bib) and a source hash; the old hand-written paper/main.tex (7 tasks, 3 models, 4 Oct; at 78bb743) is removed. - The manuscript gets Laurence's review changes already in the README: ~200-word abstract; intro as problem, state of the art, then what we do; paper-only throughout; Figure 1 legend A/B; refusals not called false positives; simulator pass rate qualified by the prompt; fidelity as the item that matters most; RNA full recovery as poor OT-2 practice; no "gaps under 0.1" claim; limitations on harness, prompt, fixed decks, published code. Grader-fault table and Docker/Modal timings move to Appendices A and B. - Superseded 4 Oct study pages (docs/harbor-results/, results/paper.html) moved to results/archive/; archive README updated. make_figures.py reads its data from there. - Corpus counts corrected from sources/master.csv: 48 papers, 166 experiments (README, Figure 1 and Figure B1 said 36/123). Figure B1 regenerated. Related work - 10 references added, each checked against its arXiv, bioRxiv or publisher page: Inagaki et al. 2023 (first GPT-4 OT-2 scripts), CLAIRify, PyLabRobot, BioLP-bench, AutoLabs, ProtoAct, LabGuard, LabSafety Bench, LABSHIELD, WetRobo. - Table 1 (manuscript) / Table F1 (README): what each of 18 related works covers against Text2WetLab (what is scored, whether it is run, what the agent works from, what it is graded against, safety checks, whether the grader is attacked, size). Drift - tests/test_docs_sync.py: trap, task, attack and corpus counts match the data in both documents; headline means match the run; the coverage table is identical in both; every citation is in refs.bib; no links to archived artefacts; the preprint outputs were built from the current source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open
- References pass the deterministic checks; with the judge the RNA references score 0.69 (abstract said "every reference solution passes"). - Not every task comes from a paper with published code: four do, one is published code without a paper, two we wrote. - Discussion now opens by placing Text2WetLab in the field and ends with what it enables and what comes next. - Non-Claude judge stated as the next step. - Reproducing: running the tasks needs only Harbor; what uv sync, OT_VENV and matplotlib are for. - Table E1: DeepSeek's run-time mean (0.581) shown, with 0.735 after the two grader fixes its run exposed. - How candidate papers and the four paper-only tasks were chosen, in both documents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Superseded: these commits are now on #61 (Laurence's branch), where the review checklist lives. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses every point in @lLegon's review on #61. A point-by-point reply is posted on #61, and #61 can be closed once this is merged.
What changes
Data fix (spotted in the review). Sonnet, Opus, Fable and GPT's RNA-easy grader records predated the
recover_about_80ulcheck. That is why they showed 32 RNA run-log checks against Qwen and DeepSeek's 33. I re-ran the checks on the saved protocols with Opentrons 7.5.0; Qwen's re-run reproduces its stored record exactly. All four models pass the check, and no reward changes. Sonnet's deterministic sub-score goes from 0.4706 to 0.4722.README and paper (both now say the same thing)
-hardsuffix.opentrons_simulate.One source for the paper.
manuscript/main.mdis the only hand-written version.build.shgenerates the HTML preprint, the PDF andmanuscript/main.tex(natbib) from it. The stale hand-writtenpaper/main.tex(7 tasks, 3 models) is removed; it's in history at78bb743. The superseded 4 October pages are moved toresults/archive/.Corpus counts. The source corpus is 48 papers and 166 experiments, per
sources/master.csv. The README, Figure 1 and Figure B1 said 36 and 123; all are corrected and the figure regenerated.Related work. 10 references are added, each checked against its arXiv, bioRxiv or publisher page:
A new coverage table (paper Table 1, README Table F1) sets 18 related works against what Text2WetLab covers.
Drift test.
tests/test_docs_sync.pyfails CI when the README and the paper disagree with the data or with each other. It checks:refs.bib;I deliberately broke the README and confirmed the test fails.
Tests
All tests pass with the Opentrons simulator: 222 passed, 2 skipped (the two skips need paper text that isn't redistributable). The site builds.
Not in this PR
main.texhas not been compiled, because there is no LaTeX installation here.🤖 Generated with Claude Code