Skip to content

docs: respond to external expert review — calibrate discovery probability - #57

Merged
cschanhniem merged 2 commits into
mainfrom
feat/external-review-response
Jun 28, 2026
Merged

cschanhniem merged 2 commits into
mainfrom
feat/external-review-response

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

Addresses the external expert review (scored 24/100) that identified three critical gaps in the pre-wet-lab package:

  • Candidate correlation not modelled: near-seed variants are correlated; if a seed family fails, most variants from it fail together. Effective independent experiments ≈ 8 scaffold families, not 20 candidates.
  • Probability estimate too optimistic: internal 29–49% does not account for correlation. Calibrated estimate: 15–30% for ≥1 hit with acceptable selectivity.
  • Competitive landscape: AMPGAN v3 (arXiv 2606.17127, June 2026) and ESCAPE benchmark (arXiv 2511.04814) set a new bar for the field.

Changes

docs/DISCOVERY_PREDICTION.md

  • New External Calibration section with honest re-estimates and explanation of near-seed correlation
  • Competitive landscape notes (AMPGAN v3, ESCAPE)
  • Benchmark limitations (AUROC measured on 44+44 demo set, not APD3-scale)
  • Novelty verification gap (45-sequence refs only)
  • Concrete path-to-probability-uplift table (6 actions × expected impact × timing)

docs/ROADMAP.md

  • Fixed stale "29–49%" to clarify it is the uncorrected internal estimate
  • Added expert review template reference to v1.0 items
  • New Beyond v1.0 gap table covering 7 items needed for a major breakthrough

.github/ISSUE_TEMPLATE/expert_review.yml

  • Structured GitHub issue template for domain-expert / wet-lab collaborator sign-off
  • Sections: sequence review, target organism + assay plan, safety/dual-use, computational score assessment, synthesis prioritisation, overall go/no-go
  • Required attestation checkboxes before synthesis order

Test plan

  • 1172 passed in 0.90s — no regressions
  • Pre-push CI hook passed
  • Human expert reads External Calibration section and confirms tone is honest and useful
  • Verify expert_review.yml renders correctly in GitHub Issues UI

🤖 Generated with Claude Code

…templates

- DISCOVERY_PREDICTION.md: add External Calibration section with honest
  re-estimate (15–30% accounting for near-seed correlation vs internal 29–49%),
  competitive landscape (AMPGAN v3, ESCAPE benchmark), benchmark limitations,
  and concrete path-to-uplift table
- ROADMAP.md: fix stale 29-49% reference, add expert review template note,
  add 'Beyond v1.0' gap table covering large-scale benchmark, external
  predictor adapters, true novelty check, and publication track
- .github/ISSUE_TEMPLATE/expert_review.yml: structured GitHub issue template
  for domain-expert / wet-lab collaborator sign-off before synthesis
- Replace incorrect '8 scaffolds × 40-60% per-scaffold rate' claim
  (that math gives >99%, not 15-30%)
- Add explicit derivation: per-scaffold all-gates rate 1.6-3.1% from
  original model × 8 effective independent scaffolds → P(≥1) ≈ 12-22%,
  rounded to 15-30% accounting for model uncertainty
- Add forward-reference to External Calibration in executive summary
- Flag AMPGAN v3 arXiv ID as requiring confirmation before external citation

Fixes HIGH issue from code-reviewer on PR #57
@cschanhniem
cschanhniem merged commit 9aa2816 into main Jun 28, 2026
3 checks passed
@cschanhniem
cschanhniem deleted the feat/external-review-response branch June 28, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant