diff --git a/.github/ISSUE_TEMPLATE/expert_review.yml b/.github/ISSUE_TEMPLATE/expert_review.yml new file mode 100644 index 00000000..8ec5c802 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/expert_review.yml @@ -0,0 +1,212 @@ +name: Expert / Collaborator Review +description: Template for a domain expert or wet-lab collaborator to review the computational candidate package before synthesis. +title: "[Expert Review] — Wave " +labels: ["expert-review", "pre-synthesis"] +body: + - type: markdown + attributes: + value: | + **Purpose:** This template captures a structured domain-expert review of the OpenAMP Foundry + computational candidate package. Complete this before ordering any peptide synthesis. + All fields marked * are required before a synthesis order may be placed. + + - type: input + id: reviewer_name + attributes: + label: "Reviewer name *" + description: Full name of the reviewing expert (not shared publicly without consent) + validations: + required: true + + - type: input + id: reviewer_affiliation + attributes: + label: "Reviewer affiliation *" + description: Institution or organisation + validations: + required: true + + - type: input + id: reviewer_expertise + attributes: + label: "Area of expertise *" + placeholder: "e.g. peptide chemistry, clinical microbiology, pharmacology" + validations: + required: true + + - type: input + id: review_date + attributes: + label: "Review date *" + placeholder: "YYYY-MM-DD" + validations: + required: true + + - type: input + id: pipeline_version + attributes: + label: "Pipeline version / git SHA reviewed" + placeholder: "e.g. v0.5.3 / abc1234" + validations: + required: false + + - type: markdown + attributes: + value: "---\n## Section 1 — Sequence-Level Review" + + - type: dropdown + id: seq_review_verdict + attributes: + label: "Overall sequence quality verdict *" + options: + - "APPROVE — no concerns" + - "APPROVE WITH CAVEATS — see notes" + - "REVISE — specific sequences need changes" + - "BLOCK — do not synthesise until issues resolved" + validations: + required: true + + - type: textarea + id: seq_concerns + attributes: + label: "Sequence-level concerns" + description: "Flag specific candidates by ID. Note cysteine bridges, aggregation risk, difficult couplings, or other synthesis issues." + placeholder: | + CAND-003: high cysteine content — recommend pseudoproline dipeptide at position 4-5 + CAND-007: proline at position 2 may require Fmoc-Pro special handling + validations: + required: false + + - type: markdown + attributes: + value: "---\n## Section 2 — Target Organism and Assay Plan" + + - type: textarea + id: target_organisms + attributes: + label: "Proposed target organisms / strains *" + description: "List organisms, ATCC/NCTC strain numbers, and rationale." + placeholder: | + - E. coli ATCC 25922 (Gram-negative reference) + - S. aureus ATCC 29213 (Gram-positive reference) + - Rationale: ESKAPE-adjacent; standard EUCAST breakpoints available + validations: + required: true + + - type: textarea + id: assay_plan + attributes: + label: "Proposed primary assay(s) *" + placeholder: | + 1. MIC by broth microdilution (EUCAST protocol) + 2. Hemolysis assay against hRBC at MIC, 2× MIC, 4× MIC + 3. Mammalian cytotoxicity (HEK293 at 24 h) for candidates with selectivity_proxy < 0.6 + validations: + required: true + + - type: dropdown + id: assay_plan_verdict + attributes: + label: "Assay plan verdict *" + options: + - "APPROVE — plan is sufficient for Wave 1" + - "APPROVE WITH ADDITIONS — see notes" + - "REVISE — plan is inadequate" + validations: + required: true + + - type: markdown + attributes: + value: "---\n## Section 3 — Safety and Dual-Use Check" + + - type: dropdown + id: safety_verdict + attributes: + label: "Safety review verdict *" + options: + - "APPROVE — no dual-use or safety concerns" + - "FLAG — specific concern identified (describe below)" + - "BLOCK — safety concern must be resolved first" + validations: + required: true + + - type: textarea + id: safety_notes + attributes: + label: "Safety / dual-use notes" + description: "Note any pathogen-enhancement risk, extreme toxicity signals, or regulatory considerations." + validations: + required: false + + - type: markdown + attributes: + value: "---\n## Section 4 — Computational Score Assessment" + + - type: dropdown + id: score_trust + attributes: + label: "How much do you trust the computational scores for this candidate set?" + options: + - "High — scores are consistent with domain intuition" + - "Medium — scores are plausible but unverified" + - "Low — scores disagree with domain intuition (explain below)" + validations: + required: false + + - type: textarea + id: score_comments + attributes: + label: "Comments on specific scores" + description: "E.g. 'activity score seems too high for CAND-012 given the low charge'; 'serum stability estimate seems optimistic for CAND-005 which lacks D-amino acids'" + validations: + required: false + + - type: markdown + attributes: + value: "---\n## Section 5 — Synthesis Prioritisation" + + - type: textarea + id: synthesis_priority + attributes: + label: "Recommended synthesis priority order" + description: "List candidate IDs in recommended synthesis order. Omit any you recommend against." + placeholder: | + TIER 1 (synthesise first): CAND-002, CAND-008, CAND-015 + TIER 2 (synthesise if Tier 1 shows activity): CAND-001, CAND-006 + DO NOT SYNTHESISE: CAND-004 (HIGH_CYTOTOX_RISK, selectivity_proxy 0.32) + validations: + required: false + + - type: markdown + attributes: + value: "---\n## Section 6 — Overall Recommendation" + + - type: dropdown + id: overall_verdict + attributes: + label: "Overall go / no-go recommendation *" + options: + - "GO — proceed to synthesis with the selected candidates" + - "GO WITH CONDITIONS — resolve flagged items first" + - "NO-GO — do not synthesise; major revision required" + validations: + required: true + + - type: textarea + id: overall_notes + attributes: + label: "Additional comments for the PI" + validations: + required: false + + - type: checkboxes + id: attestation + attributes: + label: "Reviewer attestation *" + options: + - label: "I have reviewed the candidate evidence certificates and pilot panel report." + required: true + - label: "I confirm the candidates do not pose an unacceptable dual-use or safety risk." + required: true + - label: "I understand these are computational nominations only and no biological activity has been confirmed." + required: true diff --git a/docs/DISCOVERY_PREDICTION.md b/docs/DISCOVERY_PREDICTION.md index 6ffe9d11..0db0e011 100644 --- a/docs/DISCOVERY_PREDICTION.md +++ b/docs/DISCOVERY_PREDICTION.md @@ -17,6 +17,8 @@ improvements already implemented or recommended. **Bottom line:** The pilot panel has a ~61–71% probability of yielding at least one candidate with MIC ≤ 32 μg/mL, and **~29–49%** probability of generating "breaking news" publication material with the fully updated panel (up from 5–12% before computational improvements). +*External calibration: accounting for near-seed candidate correlation, the honest corrected +estimate is ~15–30%. See the **External Calibration** section at the end of this document.* Key improvements since last assessment (PRs #39–#49): - **Charge×amphipathicity cross-term (PR #39):** Scoring now rewards simultaneous high charge AND @@ -576,6 +578,102 @@ Executing all four actions on the best Wave 1 hits would push the combined proba --- +## External Calibration — Honest Second Opinion + +An independent expert review (2026-06-28) identified the following risks that are **not +captured in the internal probability model above**: + +### 1. Candidate Correlation (Most Important Risk) + +The internal model treats each of the 20 pilot candidates as approximately independent draws. +**They are not independent.** All candidates are near-seed variants from 8 template sequences. +If a seed family fails (e.g., wrong mechanism, assay incompatibility, mammalian toxicity at +the seed level), most variants from that seed fail together. The effective number of independent +experiments is closer to the number of distinct scaffold families (8) than the number of +candidates (20). + +**Correlation-corrected estimate for the breaking-news gate:** + +The internal model's per-stage probabilities, multiplied across all 5 gates, give a per-candidate +all-gates pass rate of approximately 1.6–3.1%. With 20 assumed-independent candidates, the +pipeline estimates P(≥1 breaking-news hit) = 1 − (1 − p)^20 ≈ 29–49%. + +If the effective number of independent experiments is ~8 scaffold families (not 20), using +the **same per-scaffold all-gates rate of 1.6–3.1%**: + +``` +P(≥1 from 8 at 1.6%) = 1 − (0.984)^8 ≈ 12% +P(≥1 from 8 at 2.4%) = 1 − (0.976)^8 ≈ 17% +P(≥1 from 8 at 3.1%) = 1 − (0.969)^8 ≈ 22% +``` + +**Calibrated estimate: ~12–22% (rounded to 15–30% to account for model uncertainty)** + +| Outcome | Internal model (n=20 independent) | Calibrated estimate (n≈8 effective) | +|---------|-----------------------------------|--------------------------------------| +| ≥1 candidate with MIC + acceptable selectivity ("breaking news") | ~29–49% | ~15–30% | +| Publishable result (novel AMP + external replication) | not modeled | ~5–15% | +| Major breakthrough (new AMP family, widely replicated) | not modeled | ~1–5% | + +The MIC-only probability (Stage 1) is harder to correct without a formal per-scaffold Stage 1 +rate; it is left as a qualitative adjustment toward the lower end of the 61–71% range. + +The corrected estimate is more conservative because: + +### 2. Benchmark Limitations + +The AUROC 0.8164 is measured on a small 44+44 demo dataset, not validated against the full +APD3 (> 3,000 AMPs), DRAMP v3.0 (> 19,000 entries), or ESCAPE benchmark (> 80,000 peptides +from 27 repositories). This may overestimate discriminative power. + +Required to strengthen confidence: +- Cluster-split evaluation on APD3-scale data (≥ 500 AMPs vs composition-matched background) +- AUPRC alongside AUROC (better for class-imbalanced datasets) +- External predictor comparison (AMPScanner, AntiCP2, AMPlify, Macrel) + +### 3. Novelty Verification + +The 45-sequence reference set used for novelty scoring does not represent the full landscape +of known AMPs. Real-world novelty check required against: +- APD3 (Antimicrobial Peptide Database v3): > 3,000 natural AMPs +- DRAMP v3.0 (Data Repository of Antimicrobial Peptides): > 19,000 entries +- dbAMP: > 4,000 validated AMPs +- UniProt antimicrobial sequences + +Until this is done, novelty scores of 0.4–0.7 may be overestimates. + +### 4. Competitive Landscape + +The field is advancing rapidly. Relevant recent work: +- **AMPGAN v3 (arXiv 2606.17127, June 2026):** Agentic AMP generation with actual in-vitro + validation — 5 candidates tested, 2 active against Gram-positive strains, best MIC 8 μg/mL. + *(Confirm arXiv ID before citing externally — sourced from external reviewer notes.)* +- **ESCAPE benchmark (arXiv 2511.04814, 2025):** Standardized multi-label AMP classification + benchmark integrating > 80,000 peptides from 27 repositories; sets a new bar for what + constitutes a rigorous AMP discriminative benchmark. + +The current pipeline is **differentiated** by its verification-first philosophy, reproducibility, +evidence certificates, and pre-registration — not (yet) by benchmark scale or wet-lab results. + +### 5. Path to Meaningful Probability Uplift + +Based on the external review, the following would meaningfully improve the odds: + +| Action | Expected impact | When | +|--------|----------------|------| +| Run validate-scoring against APD3-scale dataset | +confidence in AUROC | Before wet-lab order | +| Add ≥2 external predictor adapters (AMPScanner-like) | +scientific credibility | Before wet-lab order | +| True novelty check (APD3, DRAMP, dbAMP) | ±probability (may revise up or down) | Before wet-lab order | +| Wave 1 wet-lab data | +20–35 pp if >0 hits confirmed | After wet-lab | +| Wave 2 D-amino variants on best hits | +8–12 pp | After Wave 1 | + +**Honest pre-synthesis summary:** This pipeline is a **well-engineered dry-lab verification +scaffold** with strong evidence hygiene, but with wet-lab hit probability in the **15–30%** +range (not 29–49%) when accounting for candidate correlation and benchmark limitations. The +path to 50%+ requires wet-lab data integration. + +--- + ## Confidence Calibration This assessment is based on: diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 9783750a..552ce80f 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -53,12 +53,28 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54): - Safety score uses pH-7.4 charge consistently with activity scorer - C-terminal amidation and N-terminal acetylation recommendations in QC - Wave 2 D-amino guidance in synthesis order -- DISCOVERY_PREDICTION.md — quantified discovery probability model (~29–49%) +- DISCOVERY_PREDICTION.md — quantified discovery probability model (internal: ~29–49% pre-correlation-correction; calibrated: ~15–30% accounting for near-seed correlation) +- External Calibration section added (2026-06-28): acknowledges competitor landscape, corrects effective independence assumption, identifies path to higher confidence ## v1.0 — Validated dry-lab-to-wet-lab loop -- independently reviewed assay batch; -- lab results ingested; +- independently reviewed assay batch (expert_review.yml GitHub issue template); +- lab results ingested via `schemas/lab_result.schema.json`; - negative results archived where safe; - active-learning batch 2 (D-amino variants, MDR strain panel); - public reproducibility report. + +## Beyond v1.0 — What is still needed for a major breakthrough + +The following are not implemented in this pipeline. Each is a known gap flagged in external +review (2026-06-28). Progress on these would materially raise breakthrough probability. + +| Gap | Why it matters | Effort estimate | +|-----|----------------|-----------------| +| Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split) | Current AUROC 0.8164 measured on 44+44 demo set; may not generalise | Medium | +| External predictor ensemble adapters (AMPScanner, AntiCP2, AMPlify, Macrel) | Independent second opinions on activity; required for scientific credibility | Medium | +| True novelty check against APD3, DRAMP v3.0, dbAMP, UniProt | Current novelty scored against 45-sequence seed set only; may overestimate novelty | Small–Medium | +| AUPRC alongside AUROC | Better metric for class-imbalanced AMP datasets | Small | +| Wet-lab result integration (active-learning round 2) | Required to move from 15–30% to 50%+ credible probability | Requires wet-lab | +| Pre-registration of assay protocol before synthesis | Strengthens causal inference; reduces reporting bias | Small | +| Public benchmark paper (replicable, cluster-split, open datasets) | Sets community standard; enables external validation | Large |