docs: respond to external expert review — calibrate discovery probability

.github/ISSUE_TEMPLATE/expert_review.yml

@@ -0,0 +1,212 @@
+name: Expert / Collaborator Review
+description: Template for a domain expert or wet-lab collaborator to review the computational candidate package before synthesis.
+title: "[Expert Review] <Lab / Collaborator name> — Wave <N>"
+labels: ["expert-review", "pre-synthesis"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        **Purpose:** This template captures a structured domain-expert review of the OpenAMP Foundry
+        computational candidate package. Complete this before ordering any peptide synthesis.
+        All fields marked * are required before a synthesis order may be placed.
+
+  - type: input
+    id: reviewer_name
+    attributes:
+      label: "Reviewer name *"
+      description: Full name of the reviewing expert (not shared publicly without consent)
+    validations:
+      required: true
+
+  - type: input
+    id: reviewer_affiliation
+    attributes:
+      label: "Reviewer affiliation *"
+      description: Institution or organisation
+    validations:
+      required: true
+
+  - type: input
+    id: reviewer_expertise
+    attributes:
+      label: "Area of expertise *"
+      placeholder: "e.g. peptide chemistry, clinical microbiology, pharmacology"
+    validations:
+      required: true
+
+  - type: input
+    id: review_date
+    attributes:
+      label: "Review date *"
+      placeholder: "YYYY-MM-DD"
+    validations:
+      required: true
+
+  - type: input
+    id: pipeline_version
+    attributes:
+      label: "Pipeline version / git SHA reviewed"
+      placeholder: "e.g. v0.5.3 / abc1234"
+    validations:
+      required: false
+
+  - type: markdown
+    attributes:
+      value: "---\n## Section 1 — Sequence-Level Review"
+
+  - type: dropdown
+    id: seq_review_verdict
+    attributes:
+      label: "Overall sequence quality verdict *"
+      options:
+        - "APPROVE — no concerns"
+        - "APPROVE WITH CAVEATS — see notes"
+        - "REVISE — specific sequences need changes"
+        - "BLOCK — do not synthesise until issues resolved"
+    validations:
+      required: true
+
+  - type: textarea
+    id: seq_concerns
+    attributes:
+      label: "Sequence-level concerns"
+      description: "Flag specific candidates by ID. Note cysteine bridges, aggregation risk, difficult couplings, or other synthesis issues."
+      placeholder: |
+        CAND-003: high cysteine content — recommend pseudoproline dipeptide at position 4-5
+        CAND-007: proline at position 2 may require Fmoc-Pro special handling
+    validations:
+      required: false
+
+  - type: markdown
+    attributes:
+      value: "---\n## Section 2 — Target Organism and Assay Plan"
+
+  - type: textarea
+    id: target_organisms
+    attributes:
+      label: "Proposed target organisms / strains *"
+      description: "List organisms, ATCC/NCTC strain numbers, and rationale."
+      placeholder: |
+        - E. coli ATCC 25922 (Gram-negative reference)
+        - S. aureus ATCC 29213 (Gram-positive reference)
+        - Rationale: ESKAPE-adjacent; standard EUCAST breakpoints available
+    validations:
+      required: true
+
+  - type: textarea
+    id: assay_plan
+    attributes:
+      label: "Proposed primary assay(s) *"
+      placeholder: |
+        1. MIC by broth microdilution (EUCAST protocol)
+        2. Hemolysis assay against hRBC at MIC, 2× MIC, 4× MIC
+        3. Mammalian cytotoxicity (HEK293 at 24 h) for candidates with selectivity_proxy < 0.6
+    validations:
+      required: true
+
+  - type: dropdown
+    id: assay_plan_verdict
+    attributes:
+      label: "Assay plan verdict *"
+      options:
+        - "APPROVE — plan is sufficient for Wave 1"
+        - "APPROVE WITH ADDITIONS — see notes"
+        - "REVISE — plan is inadequate"
+    validations:
+      required: true
+
+  - type: markdown
+    attributes:
+      value: "---\n## Section 3 — Safety and Dual-Use Check"
+
+  - type: dropdown
+    id: safety_verdict
+    attributes:
+      label: "Safety review verdict *"
+      options:
+        - "APPROVE — no dual-use or safety concerns"
+        - "FLAG — specific concern identified (describe below)"
+        - "BLOCK — safety concern must be resolved first"
+    validations:
+      required: true
+
+  - type: textarea
+    id: safety_notes
+    attributes:
+      label: "Safety / dual-use notes"
+      description: "Note any pathogen-enhancement risk, extreme toxicity signals, or regulatory considerations."
+    validations:
+      required: false
+
+  - type: markdown
+    attributes:
+      value: "---\n## Section 4 — Computational Score Assessment"
+
+  - type: dropdown
+    id: score_trust
+    attributes:
+      label: "How much do you trust the computational scores for this candidate set?"
+      options:
+        - "High — scores are consistent with domain intuition"
+        - "Medium — scores are plausible but unverified"
+        - "Low — scores disagree with domain intuition (explain below)"
+    validations:
+      required: false
+
+  - type: textarea
+    id: score_comments
+    attributes:
+      label: "Comments on specific scores"
+      description: "E.g. 'activity score seems too high for CAND-012 given the low charge'; 'serum stability estimate seems optimistic for CAND-005 which lacks D-amino acids'"
+    validations:
+      required: false
+
+  - type: markdown
+    attributes:
+      value: "---\n## Section 5 — Synthesis Prioritisation"
+
+  - type: textarea
+    id: synthesis_priority
+    attributes:
+      label: "Recommended synthesis priority order"
+      description: "List candidate IDs in recommended synthesis order. Omit any you recommend against."
+      placeholder: |
+        TIER 1 (synthesise first): CAND-002, CAND-008, CAND-015
+        TIER 2 (synthesise if Tier 1 shows activity): CAND-001, CAND-006
+        DO NOT SYNTHESISE: CAND-004 (HIGH_CYTOTOX_RISK, selectivity_proxy 0.32)
+    validations:
+      required: false
+
+  - type: markdown
+    attributes:
+      value: "---\n## Section 6 — Overall Recommendation"
+
+  - type: dropdown
+    id: overall_verdict
+    attributes:
+      label: "Overall go / no-go recommendation *"
+      options:
+        - "GO — proceed to synthesis with the selected candidates"
+        - "GO WITH CONDITIONS — resolve flagged items first"
+        - "NO-GO — do not synthesise; major revision required"
+    validations:
+      required: true
+
+  - type: textarea
+    id: overall_notes
+    attributes:
+      label: "Additional comments for the PI"
+    validations:
+      required: false
+
+  - type: checkboxes
+    id: attestation
+    attributes:
+      label: "Reviewer attestation *"
+      options:
+        - label: "I have reviewed the candidate evidence certificates and pilot panel report."
+          required: true
+        - label: "I confirm the candidates do not pose an unacceptable dual-use or safety risk."
+          required: true
+        - label: "I understand these are computational nominations only and no biological activity has been confirmed."
+          required: true

docs/DISCOVERY_PREDICTION.md

@@ -17,6 +17,8 @@ improvements already implemented or recommended.
 **Bottom line:** The pilot panel has a ~61–71% probability of yielding at least one candidate
 with MIC ≤ 32 μg/mL, and **~29–49%** probability of generating "breaking news" publication
 material with the fully updated panel (up from 5–12% before computational improvements).
+*External calibration: accounting for near-seed candidate correlation, the honest corrected
+estimate is ~15–30%. See the **External Calibration** section at the end of this document.*
 
 Key improvements since last assessment (PRs #39–#49):
 - **Charge×amphipathicity cross-term (PR #39):** Scoring now rewards simultaneous high charge AND
@@ -576,6 +578,102 @@ Executing all four actions on the best Wave 1 hits would push the combined proba
 
 ---
 
+## External Calibration — Honest Second Opinion
+
+An independent expert review (2026-06-28) identified the following risks that are **not
+captured in the internal probability model above**:
+
+### 1. Candidate Correlation (Most Important Risk)
+
+The internal model treats each of the 20 pilot candidates as approximately independent draws.
+**They are not independent.** All candidates are near-seed variants from 8 template sequences.
+If a seed family fails (e.g., wrong mechanism, assay incompatibility, mammalian toxicity at
+the seed level), most variants from that seed fail together. The effective number of independent
+experiments is closer to the number of distinct scaffold families (8) than the number of
+candidates (20).
+
+**Correlation-corrected estimate for the breaking-news gate:**
+
+The internal model's per-stage probabilities, multiplied across all 5 gates, give a per-candidate
+all-gates pass rate of approximately 1.6–3.1%. With 20 assumed-independent candidates, the
+pipeline estimates P(≥1 breaking-news hit) = 1 − (1 − p)^20 ≈ 29–49%.
+
+If the effective number of independent experiments is ~8 scaffold families (not 20), using
+the **same per-scaffold all-gates rate of 1.6–3.1%**:
+
+```
+P(≥1 from 8 at 1.6%) = 1 − (0.984)^8 ≈ 12%
+P(≥1 from 8 at 2.4%) = 1 − (0.976)^8 ≈ 17%
+P(≥1 from 8 at 3.1%) = 1 − (0.969)^8 ≈ 22%
+```
+
+**Calibrated estimate: ~12–22% (rounded to 15–30% to account for model uncertainty)**
+
+| Outcome | Internal model (n=20 independent) | Calibrated estimate (n≈8 effective) |
+|---------|-----------------------------------|--------------------------------------|
+| ≥1 candidate with MIC + acceptable selectivity ("breaking news") | ~29–49% | ~15–30% |
+| Publishable result (novel AMP + external replication) | not modeled | ~5–15% |
+| Major breakthrough (new AMP family, widely replicated) | not modeled | ~1–5% |
+
+The MIC-only probability (Stage 1) is harder to correct without a formal per-scaffold Stage 1
+rate; it is left as a qualitative adjustment toward the lower end of the 61–71% range.
+
+The corrected estimate is more conservative because:
+
+### 2. Benchmark Limitations
+
+The AUROC 0.8164 is measured on a small 44+44 demo dataset, not validated against the full
+APD3 (> 3,000 AMPs), DRAMP v3.0 (> 19,000 entries), or ESCAPE benchmark (> 80,000 peptides
+from 27 repositories). This may overestimate discriminative power.
+
+Required to strengthen confidence:
+- Cluster-split evaluation on APD3-scale data (≥ 500 AMPs vs composition-matched background)
+- AUPRC alongside AUROC (better for class-imbalanced datasets)
+- External predictor comparison (AMPScanner, AntiCP2, AMPlify, Macrel)
+
+### 3. Novelty Verification
+
+The 45-sequence reference set used for novelty scoring does not represent the full landscape
+of known AMPs. Real-world novelty check required against:
+- APD3 (Antimicrobial Peptide Database v3): > 3,000 natural AMPs
+- DRAMP v3.0 (Data Repository of Antimicrobial Peptides): > 19,000 entries
+- dbAMP: > 4,000 validated AMPs
+- UniProt antimicrobial sequences
+
+Until this is done, novelty scores of 0.4–0.7 may be overestimates.
+
+### 4. Competitive Landscape
+
+The field is advancing rapidly. Relevant recent work:
+- **AMPGAN v3 (arXiv 2606.17127, June 2026):** Agentic AMP generation with actual in-vitro
+  validation — 5 candidates tested, 2 active against Gram-positive strains, best MIC 8 μg/mL.
+  *(Confirm arXiv ID before citing externally — sourced from external reviewer notes.)*
+- **ESCAPE benchmark (arXiv 2511.04814, 2025):** Standardized multi-label AMP classification
+  benchmark integrating > 80,000 peptides from 27 repositories; sets a new bar for what
+  constitutes a rigorous AMP discriminative benchmark.
+
+The current pipeline is **differentiated** by its verification-first philosophy, reproducibility,
+evidence certificates, and pre-registration — not (yet) by benchmark scale or wet-lab results.
+
+### 5. Path to Meaningful Probability Uplift
+
+Based on the external review, the following would meaningfully improve the odds:
+
+| Action | Expected impact | When |
+|--------|----------------|------|
+| Run validate-scoring against APD3-scale dataset | +confidence in AUROC | Before wet-lab order |
+| Add ≥2 external predictor adapters (AMPScanner-like) | +scientific credibility | Before wet-lab order |
+| True novelty check (APD3, DRAMP, dbAMP) | ±probability (may revise up or down) | Before wet-lab order |
+| Wave 1 wet-lab data | +20–35 pp if >0 hits confirmed | After wet-lab |
+| Wave 2 D-amino variants on best hits | +8–12 pp | After Wave 1 |
+
+**Honest pre-synthesis summary:** This pipeline is a **well-engineered dry-lab verification
+scaffold** with strong evidence hygiene, but with wet-lab hit probability in the **15–30%**
+range (not 29–49%) when accounting for candidate correlation and benchmark limitations. The
+path to 50%+ requires wet-lab data integration.
+
+---
+
 ## Confidence Calibration
 
 This assessment is based on:

docs/ROADMAP.md

@@ -53,12 +53,28 @@ Implemented during the pre-wet-lab improvement loop (PRs #31–#54):
 - Safety score uses pH-7.4 charge consistently with activity scorer
 - C-terminal amidation and N-terminal acetylation recommendations in QC
 - Wave 2 D-amino guidance in synthesis order
-- DISCOVERY_PREDICTION.md — quantified discovery probability model (~29–49%)
+- DISCOVERY_PREDICTION.md — quantified discovery probability model (internal: ~29–49% pre-correlation-correction; calibrated: ~15–30% accounting for near-seed correlation)
+- External Calibration section added (2026-06-28): acknowledges competitor landscape, corrects effective independence assumption, identifies path to higher confidence
 
 ## v1.0 — Validated dry-lab-to-wet-lab loop
 
-- independently reviewed assay batch;
-- lab results ingested;
+- independently reviewed assay batch (expert_review.yml GitHub issue template);
+- lab results ingested via `schemas/lab_result.schema.json`;
 - negative results archived where safe;
 - active-learning batch 2 (D-amino variants, MDR strain panel);
 - public reproducibility report.
+
+## Beyond v1.0 — What is still needed for a major breakthrough
+
+The following are not implemented in this pipeline. Each is a known gap flagged in external
+review (2026-06-28). Progress on these would materially raise breakthrough probability.
+
+| Gap | Why it matters | Effort estimate |
+|-----|----------------|-----------------|
+| Large-scale benchmark (≥ 500 AMPs vs composition-matched decoys, cluster-split) | Current AUROC 0.8164 measured on 44+44 demo set; may not generalise | Medium |
+| External predictor ensemble adapters (AMPScanner, AntiCP2, AMPlify, Macrel) | Independent second opinions on activity; required for scientific credibility | Medium |
+| True novelty check against APD3, DRAMP v3.0, dbAMP, UniProt | Current novelty scored against 45-sequence seed set only; may overestimate novelty | Small–Medium |
+| AUPRC alongside AUROC | Better metric for class-imbalanced AMP datasets | Small |
+| Wet-lab result integration (active-learning round 2) | Required to move from 15–30% to 50%+ credible probability | Requires wet-lab |
+| Pre-registration of assay protocol before synthesis | Strengthens causal inference; reduces reporting bias | Small |
+| Public benchmark paper (replicable, cluster-split, open datasets) | Sets community standard; enables external validation | Large |