Review readme

[lLegon]

Reviewed readme and added comments, expand all lines on the readme in the Files Changed view to see them

[lLegon]

Points to address

[lLegon]

6 models cos I think you added quen and deepseek?

[lLegon]

Quen and deepseek have a different no of run-log checks

[lLegon]

As a broad comment this is a bit wordy for an abstract (typically ~200 words, although this depends on where it is being submitted)

[lLegon]

So an intro would usually start on the problem, a few paragraphs on the state of the art, then 1 paragraph giving a quick summary of the thing we are doing here. Different journals have different styles though so would depend on where we are aiming

[lLegon]

Would a fixed deck be a realistic scenario? Perhaps we should also evaluate the agent's ability to pick the correct deck and labware

[lLegon]

Note- we don't get 11/11 references passing when judge is active, that only refers to deterministic checks

[lLegon]

Yep we should definitely try with a non claude judge

[lLegon]

Why does deepseek not have a mean reward?

[lLegon]

We should probably expand on candidate selection or not talk about it at all

[lLegon]

A discussion usually puts the work in place within the field- discusses what else exists and how this is helping, and some future directions that this might enable

[Tyronita]

Thanks @lLegon. #64 addresses this review; here is each point and what happened to it.

Fixed in #64

Line	Your point	What changed
34	Say 6 models	The contributions now say six models from four providers, two of them open-weight
307	Qwen and DeepSeek have a different number of run-log checks	You found a real gap: the other four models' RNA records predated the 80 µL check. I re-ran it on the saved protocols. Every model is now 33/33, and no reward changes
21	Abstract too wordy	Cut to about 200 words
26	Intro structure	Now problem, then state of the art, then one paragraph on what we do
32	"4 of them built from a paper" is unclear	Now "4 paper-only tasks where the agent gets only a goal and the source paper"
40, 42	Figure only mentions Anthropic models; label panels A/B; legend unclear; filenames too detailed	Figure 1 redrawn with panels A/B, all six models and the verifier steps, and no filenames. The caption can be read without the paper
92	Too many frames	Figure 3 shows 2 labelled frames instead of 12
146	Check these are valid	Checked against every task's rubric.json: all correct. Only the wording was tidied
168	Why is the formatting different on this line?	Because "fake comments" was never a trap: there are 9 traps, not 10, and the judge fails fake comments under the fidelity item. Fixed in the README, the paper, the site and Figure 1
190	Don't describe how we fixed the code	The section keeps the validation in both directions; the fault-and-fix table moves to an appendix
282	Pick "hard" or "paper-only"	"Paper-only" throughout. Task names keep the -hard suffix
282	Restrict cost to tasks every model answered	Added to the summary table and Figure 7b: Sonnet $0.75, GPT $0.97, Fable $5.53
278	Refusals aren't false positives; Opus vs Fable on heat shock is the interesting result	Reworded as you suggested. The text now notes that Opus answered the transformation prompt Fable refused, and that one attempt can't tell a non-deterministic classifier from a per-model one
312	Is the simulator pass rate from the prompt?	Yes in part: the easy-task instructions tell the agent to run opentrons_simulate. This is now stated as a qualification and listed as a limitation
312	Fidelity to the paper is the most important thing	Now framed that way in the results
316, 324	Supplementary material	REPORT.md is labelled supplementary; the Docker vs Modal timings move to an appendix, with the hardware stated in the methods
319	Recovering 100 µL is still wrong on an OT-2	Agreed. It is now described as poor OT-2 practice, and the paper-only rubric not failing it is listed as a limitation (robot_practice should fail it)
356	Source for "gaps under 0.1 are not meaningful"?	There isn't one, so the claim is removed. The text now says one attempt is not enough to rank close scores, and notes the non-linearity you described: the critical cap turns one check into a drop of 0.7, while an uncaught mistake may cost a single rubric item

Listed as limitations; they need new runs

• Line 29, fixed decks: a fixed deck isn't realistic, and choosing the deck and labware should also be evaluated.
• Line 29, published vs unpublished code: published code risks training leakage and may come from authors who report procedures more fully.
• Line 34, harness bias: Anthropic models are probably better tuned to Claude Code.
• Line 34, prompt ablation: compare with and without the prompt guidance.
• Line 235, repeats: these would give error bars, confidence intervals and significance tests.

These are worth doing; I'd suggest repeats (-k 3) first, since they're the cheapest.

Lines 322 and 353: no change needed for 322 ("doesn't matter too much"). For 353, agreed: access to bio tasks would let us score the refused ones.

Also in #64

• Related work: 10 verified references added, including Inagaki et al. 2023 (the first GPT-4 OT-2 scripts), AutoLabs, ProtoAct, BioLP-bench, LabSafety Bench, LABSHIELD, LabGuard and WetRobo.
• Coverage table: sets 18 related works against what we cover. We're the only work that has an agent read a paper and then checks every well's end state against a ground truth. We're also small (11 tasks) and have no wet-lab runs.
• Corpus numbers: corrected to 48 papers and 166 experiments.

Proposal: a structure that can't drift

At the moment the README and the paper are both written by hand, and tests/test_docs_sync.py only detects when they disagree. To make drift impossible rather than just detected:

1. Generate every number. A script scripts/facts.py would write results/facts.json from the data: run results, adversarial.json, sources/master.csv and the trap list. It would hold the counts, headline scores, costs and the best attack score. Prose would refer to these as {{facts.headline.opus}}, and a render step would fill them in.
2. Generate every table. The summary, risk, task and fault tables come from the data. The coverage table comes from docs/related_work.yaml, which also holds each work's citation key and a one-line description.
3. Write shared text once. The abstract, the contributions, the limitations and the table captions would live in docs/shared/*.md. Both README.md and manuscript/main.md would pull them in through markers like <!-- include: abstract -->, and scripts/build_docs.py would fill them in. The README would keep only GitHub-specific parts: badges, mermaid diagrams, run commands.
4. Never hand-edit outputs. The preprint HTML, main.tex, the PDF (attached to a release), the site, and the Hugging Face card (once HF deploy mirrors the allowlist (removes 161 stale files incl. old answer keys); card generated from latest run #56 lands) are only ever built.
5. CI regenerates and diffs. It would run build_docs.py and build.sh, then git diff --exit-code. Any hand edit to a generated part fails, and the sync test becomes a by-product.
6. Archive policy. Anything superseded moves to results/archive/ with a line in its README, never left beside current results.

Steps 1 and 3 remove most of the hand-copying: every edit to this review had to be made twice. Happy to implement this as a follow-up PR if you agree.

[Tyronita]

Review checklist: all 37 comments

All the work is now on this branch (66758f2); I've closed #64. Everything is checked against origin/main as of today. ✅ = addressed in this PR; ⏳ = not done yet, because it needs a new run or a decision, and is stated in the README and paper as a limitation or next step.

Writing and structure

• ✅ Abstract too long (L21): cut to about 200 words
• ✅ 11/11 references only pass the deterministic checks (L21): the abstract now says "passes the deterministic checks"; §3.4 explains the RNA references' 0.69 with the judge
• ✅ Intro structure (L26): now problem, then state of the art, then what we do
• ✅ Not all tasks come from papers with code (L29): four do, one is published code without a paper, two we wrote; fixed in the intro and limitations of both documents
• ✅ "4 of them built from a paper" is unclear (L32): reworded
• ✅ Say 6 models (L34): six models from four providers
• ✅ Discussion should place the work in the field, plus future directions (L314): new opening paragraph on where it sits, and a closing "what it enables, and what comes next" paragraph
• ✅ Use one term, hard or paper-only (L282): "paper-only" throughout (task names keep -hard)
• ✅ Don't describe how we fixed the code (L190): the fault table moved to Appendix D
• ✅ Error analysis is supplementary material (L316): REPORT.md labelled supplementary
• ✅ Speed is supplementary; mention hardware in methods (L324): timings moved to Appendix E; hardware is in the setup table
• ✅ Expand on candidate selection or drop it (L387): expanded: PubMed Central search, conversion, and why these four papers
• ✅ Need uv sync, or just Harbor? (L334): running the tasks needs only Harbor; the README says what uv sync, OT_VENV and matplotlib are for

Figures and tables

• ✅ Subfigures A/B (L42): Figure 1 redrawn with panels A/B
• ✅ Figure only mentions Anthropic models (L42): now all six models
• ✅ Unclear legend; no mention of the verifier (L42): the legend now reads on its own and describes the verifier
• ✅ Filenames too detailed (L42): removed from the figure and caption
• ✅ Too many frames (L92): 2 labelled frames instead of 12
• ✅ Check these rubric items are valid (L148): checked against every rubric.json; all correct
• ✅ Why is the formatting different on this line? (L168): "fake comments" was never a trap; there are 9 traps, not 10, fixed everywhere
• ✅ Qwen/DeepSeek have a different number of run-log checks (L307): a real gap; the earlier four models' records are regraded, all now 33/33, and no reward changes
• ✅ Why does DeepSeek have no mean reward? (L340): shown: 0.581 as graded at run time, 0.735 after the two grader fixes its run exposed

Claims

• ✅ Refusals aren't false positives; Opus vs Fable is the interesting result (L278): reworded; the Opus/Fable inconsistency is now discussed
• ✅ Restrict cost to tasks every model answered (L282): added to the summary table and Figure 7b
• ✅ Fidelity to the paper is the most important thing (L312): framed that way in the results
• ✅ Recovering 100 µL is still wrong on an OT-2 (L319): the text now says so (rubric change below)
• ✅ Source for "gaps under 0.1 are not meaningful"? (L356): claim removed; the critical-cap non-linearity is now explained
• ✅ "Honestly doesn't matter too much" (L308): no change needed

Experiments and grading: not done yet (each is stated as a limitation or next step)

• ⏳ Fixed deck isn't realistic; evaluate deck and labware choice (L29): needs new tasks and a grader that matches labware by role
• ⏳ Leakage from published code; compare with unpublished (L29): needs unpublished protocols, from collaborating labs or written by us
• ⏳ Authors who publish code may report more detail (L29): same unpublished-protocol comparison
• ⏳ Evaluate without the harness (L34): needs new runs
• ⏳ With/without prompt engineering (L34): needs a prompt ablation run
• ⏳ Is the simulator pass rate due to the prompt? (L312): text qualified (the easy prompt says to run opentrons_simulate); the test itself needs the ablation above
• ⏳ Repeats for error bars and significance tests (L235): suggested as the next run (-k 3)
• ⏳ Try a non-Claude judge (L318): stated as the next step; needs a re-judge run
• ⏳ RNA-hard full recovery should fail robot_practice (L319): rubric change still to make
• ⏳ Get access to bio tasks from Anthropic (L353): outside the repo

Totals: 28 addressed; 10 open items covering 9 comments (the L319 comment counts once above and once here). All 10 need runs, a rubric change or outside access. A proposal to stop the README and paper drifting apart is in my earlier comment.

[Tyronita]

@lLegon could you review the fixes I've pushed to this branch? They're in four commits:

1. 34242a0 Regrades the RNA records your run-log comment caught. All six models are now 33/33, and no reward changes.
2. 0124bcb Applies your README review: abstract, intro, figures, terminology, refusals, cost, fidelity, RNA recovery, the 0.1 claim, and the limitations.
3. 4295c42 Makes manuscript/main.md the only source for the paper, adds 10 related-work references and a coverage table, and adds a drift test.
4. 66758f2 Applies your later comments: references pass the deterministic checks only, task sources, the discussion, the non-Claude judge, uv vs Harbor, DeepSeek's mean, and candidate selection.

The full checklist is in my comment above. If you're happy with the fixes, an approving review lets this merge.

Still open

None of these block the merge. Each is stated in the README and paper as a limitation or next step.

• Fixed decks aren't realistic; evaluate deck and labware choice: needs new tasks and a grader that matches labware by role
• Leakage from published code into training data: needs unpublished protocols
• Authors who publish code may report more detail: the same unpublished-protocol comparison
• Evaluate without the harness: needs new runs
• With and without prompt engineering: needs a prompt ablation
• Is the simulator pass rate due to the prompt?: the text is qualified; the test needs the same ablation
• Repeats for error bars and significance tests: a -k 3 run
• Try a non-Claude judge: a re-judge with another provider's model
• RNA-hard full recovery should fail: rubric change to robot_practice on the paper-only task, then re-judge
• Access to bio tasks from Anthropic: outside the repo

The rubric change and the repeat run are the cheapest to start. Which should go first?