diff --git a/docs/DISCOVERY_PREDICTION.md b/docs/DISCOVERY_PREDICTION.md index e7bed62e..4260b162 100644 --- a/docs/DISCOVERY_PREDICTION.md +++ b/docs/DISCOVERY_PREDICTION.md @@ -243,9 +243,9 @@ Fmoc SPPS with acetonitrile/water gradient; verify MALDI-TOF pre-assay. Updated from ~55–65% due to addition of proven scaffold families. Basis: -- Pipeline AUROC = 0.8348 (95% CI: TBD; PR #70 windowed mu_h + anionic guard; 43 unique AMPs, n=87 total; previous 0.8047 PR #66, 0.8086 pre-dedup, 0.8164 pre-Trp-bonus) -- Pipeline AUPRC = 0.8443 (+0.2943 above random baseline 0.4943; PR curve emphasises precision at selection operating point) -- Recall@20 = 43% on internal benchmark (positives recovered in top 20 ranked candidates) +- Pipeline AUROC = 0.8420 (bootstrap CI₉₅: 0.76–0.91, n_bootstrap=2000; PR #72 face_segregation_bonus; 43 unique AMPs, n=87 total; previous 0.8348 PR #70, 0.8047 PR #66, 0.8086 pre-dedup, 0.8164 pre-Trp-bonus) +- Pipeline AUPRC = 0.8627 (+0.3684 above random baseline 0.4943; PR curve emphasises precision at selection operating point) +- Recall@20 = 44% on internal benchmark (positives recovered in top 20 ranked candidates) - SEED-003 family (up to 4/20): RRWQWRMKKLG is curated known AMP → variants ~65–75% hit rate - SEED-007 family (up to 4/20): Bombolitin II (*Bombus pennsylvanicus* bumblebee venom) with antimicrobial activity documented in Bozelli et al. (2017) BBA-Biomembranes and subsequent @@ -467,14 +467,15 @@ Despite the probability gap, the pipeline has done the following correctly: 2. **Consistent synthesis feasibility:** All pilot candidates are MODERATE SPPS difficulty except one HIGH (SEED-002 23-mer). Zero synthesis-impossible candidates nominated. -3. **Strong AUROC:** 0.811 (bootstrap CI₉₅: 0.71–0.89) — the ensemble correctly separates - AMP-like from random-sequence background ~80% of the time. +3. **Strong AUROC:** 0.8420 (bootstrap CI₉₅: 0.76–0.91, n=87, n_bootstrap=2000; PR #72 face_segregation_bonus) — the ensemble correctly separates + AMP-like from random-sequence background ~84% of the time. -4. **Dual-scorer consensus:** All phase3 nominees have disagreement ≤ 0.40 (physicochemical and +4. **Dual-scorer consensus:** All phase3 nominees have disagreement ≤ 0.45 (physicochemical and Boman scorers agree within the validated threshold). Note: SEED-008 (Trp-rich puroindoline-a) - class sequences produce disagreement ~0.37 — this reflects Boman scale's W=-3.398 limitation + class sequences produce disagreement ~0.43 — this reflects Boman scale's W=-3.398 limitation (protein-binding potential) vs Trp's actual interfacial insertion mechanism. The threshold was - raised from 0.30 to 0.40 to not falsely exclude this validated AMP mechanism class (PR #61). + raised from 0.30→0.40 (PR #61) and 0.40→0.45 (PR #72) to not falsely exclude this validated + AMP mechanism class. Non-Trp-rich candidates in the 709-sequence pool remain below 0.41. 5. **Safety-first selection:** Safety ≥ 0.60 required (max_safety_risk = 0.40). Mean panel safety = 0.991. SEED-010 (KR-12 human LL-37 fragment) correctly excluded for cytotoxicity @@ -604,10 +605,10 @@ Executing all four actions on the best Wave 1 hits would push the combined proba ## Summary Table: Probability by Gate -| Stage | Gate | Original | After PRs #31–38 | After PRs #39–42 | After PRs #43–47 | After PRs #48–53 (current) | Primary limiting factor | +| Stage | Gate | Original | After PRs #31–38 | After PRs #39–42 | After PRs #43–47 | After PRs #61–72 (current) | Primary limiting factor | |-------|------|----------|-----------------|-----------------|-----------------|--------------------------|------------------------| | 0 | Synthesis success | ~90% | ~90% | ~88% | **~89%** ✓ | **~90%** ✓ (agg model + agg-safe gen + pro penalty + pH74 charge) | SEED-008 W-rich; all aggregation/synthesis risks modelled | -| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~62–72%** ✓ (AUROC 0.8348) | AUROC 0.8348 (AUPRC 0.8443); 6 scaffold families confirmed | +| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~62–72%** ✓ (AUROC 0.8420) | AUROC 0.8420 (AUPRC 0.8627); 6 scaffold families confirmed | | 2 | TI > 10 (selectivity) | ~35–50% | ~35–50% | ~38–52% | **~40–55%** ✓ | **~42–57%** ✓ (stronger SEED-004 demotion) | sel_proxy doubled penalty for HIGH_CYTOTOX_RISK tier | | 3 | t½ > 2 h (serum) | ~10–20% | ~25–40% | ~28–42% | **~28–42%** | **~30–46%** ✓ (3-protease model + short/Trp-rich model correction; pilot-panel data) | SEED-003/008 may outperform model score; all seeds borderline — early serum assay recommended | | 4 | Scaffold novelty | ~10–15% | ~18–25% | ~25–35% | **~26–36%** ✓ | **~26–36%** | Diversity filter removes cross-seed near-dups | @@ -683,7 +684,7 @@ The corrected estimate is more conservative because: ### 2. Benchmark Limitations -The AUROC 0.8348 (phase3 synthesis gate: 0.8126) is measured on a small 43+44 demo dataset, not validated against the full +The AUROC 0.8420 (phase3 synthesis gate: 0.8266) is measured on a small 43+44 demo dataset, not validated against the full APD3 (> 3,000 AMPs), DRAMP v3.0 (> 19,000 entries), or ESCAPE benchmark (> 80,000 peptides from 27 repositories). This may overestimate discriminative power. @@ -739,7 +740,7 @@ than the 10 originally planned. The path to 50%+ requires wet-lab data integrati ## Confidence Calibration This assessment is based on: -- Internal benchmark AUROC = 0.8348 (n=87, bootstrap n=2000; phase3 synthesis gate = 0.8126; PR #70 windowed mu_h + anionic guard) +- Internal benchmark AUROC = 0.8420 (n=87, bootstrap CI₉₅: 0.76–0.91, n_bootstrap=2000; phase3 synthesis gate = 0.8266; PR #72 face_segregation_bonus; PR #70 windowed mu_h + anionic guard) - Literature hit rates for physchem AMP prediction (Loose et al. 2006; Tossi et al. 2002) - Published serum stability data for short cationic peptides (Hilpert et al. 2006) - D-amino acid t½ extension data (Wade et al. 1990, PNAS)