Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/DISCOVERY_PREDICTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
**Pipeline:** OpenAMP-Foundry v0.1.0
**Date:** 2026-06-28 (updated 2026-06-28)
**Status:** Pre-synthesis scientific assessment — for expert review before ordering
**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37), SEED-006 Mastoparan-X (PR #38), Charge×amphipathicity cross-term (PR #39), Amphipathicity weight + helix_bonus (PR #40), SEED-007 Bombolitin II + SEED-008 Puroindoline-a (PR #41), N-terminal acetylation flag + D-amino Wave 2 guidance (PR #42), Full synthesis risk QC — QG/QS deamidation, DG/DS isomerization, Trp photolability (PR #44), Vendor-ready synthesis order generator (PR #45), Diversity-aware pilot panel selection similarity-threshold=0.75 (PR #46), Selectivity proxy + HIGH_CYTOTOX_RISK flag — charge/GRAVY-based mammalian cytotoxicity risk detector (PR #47), selectivity_proxy routed into pilot_priority formula — low-cytotox-risk candidates gently demoted (PR #48), Elastase resistance (HNE 3-protease stability model) + aggregation propensity scoring (synthesis feasibility penalty) (PR #49), Aggregation-safe mutation generation + balanced K/R charge variants + SynthQC continuous aggregation score (PR #50), Proline synthesis penalty + helix bonus enhancement — AUROC 0.8138 → 0.8164 (PR #51), Safety pH74 charge consistency + stronger cytotoxicity penalty — SEED-004 net selectivity swing increased 0.035→0.055 (PR #53), AUPRC metric + SEED-009 (Bac2A proline-rich) + SEED-010 (KR-12 LL-37 fragment) (PR #58), ATCC strain corrections + preregistration assay protocol + negative result schema (PR #60), **CRITICAL FIX: SEED-008 Trp-rich scaffold reinstated in synthesis pool (was excluded by Boman W scale artifact); all 10 seeds generated; SEED-007/009 newly included; SEED-004 correctly excluded by safety gate; 6 mechanism-diverse scaffolds confirmed in synthesis pool** (PR #61)
**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37), SEED-006 Mastoparan-X (PR #38), Charge×amphipathicity cross-term (PR #39), Amphipathicity weight + helix_bonus (PR #40), SEED-007 Bombolitin II + SEED-008 Puroindoline-a (PR #41), N-terminal acetylation flag + D-amino Wave 2 guidance (PR #42), Full synthesis risk QC — QG/QS deamidation, DG/DS isomerization, Trp photolability (PR #44), Vendor-ready synthesis order generator (PR #45), Diversity-aware pilot panel selection similarity-threshold=0.75 (PR #46), Selectivity proxy + HIGH_CYTOTOX_RISK flag — charge/GRAVY-based mammalian cytotoxicity risk detector (PR #47), selectivity_proxy routed into pilot_priority formula — low-cytotox-risk candidates gently demoted (PR #48), Elastase resistance (HNE 3-protease stability model) + aggregation propensity scoring (synthesis feasibility penalty) (PR #49), Aggregation-safe mutation generation + balanced K/R charge variants + SynthQC continuous aggregation score (PR #50), Proline synthesis penalty + helix bonus enhancement — AUROC 0.8138 → 0.8164 (PR #51), Safety pH74 charge consistency + stronger cytotoxicity penalty — SEED-004 net selectivity swing increased 0.035→0.055 (PR #53), AUPRC metric + SEED-009 (Bac2A proline-rich) + SEED-010 (KR-12 LL-37 fragment) (PR #58), ATCC strain corrections + preregistration assay protocol + negative result schema (PR #60), **CRITICAL FIX: SEED-008 Trp-rich scaffold reinstated in synthesis pool (was excluded by Boman W scale artifact); all 10 seeds generated; SEED-007/009 newly included; SEED-004 correctly excluded by safety gate; 6 mechanism-diverse scaffolds confirmed in synthesis pool** (PR #61), Phase3.yaml AUROC documented (0.7936→0.7890 after PR #65) (PR #63), Stale threshold/reference-count docs corrected (PR #64), **Trp-weighted aromatic bonus (1.5× Trp vs Phe/Tyr; Wimley-White interfacial mechanism); safety abs() bug fix (negative charge does not cause hemolysis); Eisenberg scale comment fix; pool SEED-006: 11→10, SEED-009: 18→19; AUROC 0.8164→0.8086** (PR #65)

---

Expand Down Expand Up @@ -243,8 +243,8 @@ Fmoc SPPS with acetonitrile/water gradient; verify MALDI-TOF pre-assay.
Updated from ~55–65% due to addition of proven scaffold families.

Basis:
- Pipeline AUROC = 0.8164 (95% CI: 0.72–0.90) vs composition-matched UniProt decoys
- Pipeline AUPRC = 0.8556 (+0.36 above random baseline 0.50; PR curve emphasises precision at selection operating point)
- Pipeline AUROC = 0.8086 (95% CI: 0.71–0.90) vs composition-matched UniProt decoys (PR #65 update; previous 0.8164 after Trp bonus scientific fix; within bootstrap CI)
- Pipeline AUPRC = 0.8495 (+0.35 above random baseline 0.50; PR curve emphasises precision at selection operating point)
- Recall@20 = 43% on internal benchmark (positives recovered in top 20 ranked candidates)
- SEED-003 family (up to 4/20): RRWQWRMKKLG is curated known AMP → variants ~65–75% hit rate
- SEED-007 family (up to 4/20): Bombolitin II (*Bombus pennsylvanicus* bumblebee venom) with
Expand Down Expand Up @@ -543,7 +543,7 @@ in the 20-member pilot panel, P(≥1 from family) ≈ 1 − (1 − P_candidate)^
| 1 | SEED-008 | FPVTWRWWKWWKG vars | 0.385 | **0.667** | ~15–25% | Highest novelty + proven Trp mechanism; 24 selected (PR #61 reinstated) |
| 2 | SEED-007 | IKITTMLKKLG vars | 0.636 | 0.615 | ~14–22% | Bumblebee venom; distinct from wasp mastoparan; 27 selected (highest count) |
| 3 | SEED-006 | INWKGIAAMAKKLL vars | 0.667 | 0.643 | ~14–22% | Balanced stability+novelty; mastoparan mechanism data |
| 4 | SEED-009 | RRLPRPPYLPRP vars | 0.80+ | 0.467 | ~12–20% | Proline-rich intracellular (DnaK); orthogonal mechanism to all other seeds; 18 selected |
| 4 | SEED-009 | RRLPRPPYLPRP vars | 0.80+ | 0.467 | ~12–20% | Proline-rich intracellular (DnaK); orthogonal mechanism to all other seeds; 19 selected |
| 5 | SEED-003 | RRWQWRMKKLG vars | 0.27 | 0.182 | ~8–16% | High ensemble (0.780-0.795) but poor serum stability; 16 selected |
| 6 | SEED-005 | KRLFKKIGSALKFL (seed) | 0.52 | 0.467 | ~8–14% | Cecropin-magainin hybrid; best variant VAR_009 (KRFFKKIGSALKFA, FF motif) aids membrane insertion; 4 selected |

Expand Down Expand Up @@ -584,7 +584,7 @@ Executing all four actions on the best Wave 1 hits would push the combined proba
| Stage | Gate | Original | After PRs #31–38 | After PRs #39–42 | After PRs #43–47 | After PRs #48–53 (current) | Primary limiting factor |
|-------|------|----------|-----------------|-----------------|-----------------|--------------------------|------------------------|
| 0 | Synthesis success | ~90% | ~90% | ~88% | **~89%** ✓ | **~90%** ✓ (agg model + agg-safe gen + pro penalty + pH74 charge) | SEED-008 W-rich; all aggregation/synthesis risks modelled |
| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~61–71%** ✓ (AUROC 0.8164) | AUROC 0.8164 (AUPRC computed); 10 scaffold families |
| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~61–71%** ✓ (AUROC 0.8086) | AUROC 0.8086 (AUPRC 0.8495); 6 scaffold families confirmed |
| 2 | TI > 10 (selectivity) | ~35–50% | ~35–50% | ~38–52% | **~40–55%** ✓ | **~42–57%** ✓ (stronger SEED-004 demotion) | sel_proxy doubled penalty for HIGH_CYTOTOX_RISK tier |
| 3 | t½ > 2 h (serum) | ~10–20% | ~25–40% | ~28–42% | **~28–42%** | **~29–44%** ✓ (3-protease model) | Wave 2 D-amino plan machine-readable |
| 4 | Scaffold novelty | ~10–15% | ~18–25% | ~25–35% | **~26–36%** ✓ | **~26–36%** | Diversity filter removes cross-seed near-dups |
Expand Down Expand Up @@ -660,7 +660,7 @@ The corrected estimate is more conservative because:

### 2. Benchmark Limitations

The AUROC 0.8164 is measured on a small 44+44 demo dataset, not validated against the full
The AUROC 0.8086 (phase3 synthesis gate: 0.7890) is measured on a small 44+44 demo dataset, not validated against the full
APD3 (> 3,000 AMPs), DRAMP v3.0 (> 19,000 entries), or ESCAPE benchmark (> 80,000 peptides
from 27 repositories). This may overestimate discriminative power.

Expand Down Expand Up @@ -716,7 +716,7 @@ than the 10 originally planned. The path to 50%+ requires wet-lab data integrati
## Confidence Calibration

This assessment is based on:
- Internal benchmark AUROC = 0.8164 (n=88, bootstrap n=2000; improved from 0.8138 after helix bonus weight PR #51)
- Internal benchmark AUROC = 0.8086 (n=88, bootstrap n=2000; phase3 synthesis gate = 0.7890; PR #65 Trp-weighted bonus update)
- Literature hit rates for physchem AMP prediction (Loose et al. 2006; Tossi et al. 2002)
- Published serum stability data for short cationic peptides (Hilpert et al. 2006)
- D-amino acid t½ extension data (Wade et al. 1990, PNAS)
Expand All @@ -739,8 +739,8 @@ is the fastest path from the current ~10–18% (calibrated) to the ≥20–30% t
justify a larger Wave 2 investment.

**Synthesis pool status (PR #61, final pre-wetlab state):**
6 scaffold families confirmed: SEED-003 (16), SEED-005 (4), SEED-006 (11), SEED-007 (27),
SEED-008 (24), SEED-009 (18). Total: 100 selected candidates. Mean safety=0.991, mean
6 scaffold families confirmed: SEED-003 (16), SEED-005 (4), SEED-006 (10), SEED-007 (27),
SEED-008 (24), SEED-009 (19). Total: 100 selected candidates. Mean safety=0.991, mean
ensemble=0.805. 20-member pilot panel drawn from these 100 for Wave 1 synthesis.

---
Expand Down
33 changes: 17 additions & 16 deletions docs/METHODS.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,29 +316,29 @@ Phase 2 benchmarks verified that the pipeline:
- Produces stable rankings under repeated runs (reproducibility)
- Shows performance degradation when key scoring dimensions are ablated

**Retrospective AUROC — pipeline.yaml (v0.2.x):** `0.8164` (positive-vs-negative separation on
the 44-AMP + 44-background benchmark set using the full ensemble scorer with pipeline.yaml
weights). Benchmarked after PRs #48–54; the scoring improvements in those PRs moved AUROC from
0.7926 → 0.8138 → 0.8164. Bootstrap 95% CI computed via Wilcoxon-Mann-Whitney (n=2000
iterations; CI reported in `validate-scoring` output).

**Retrospective AUROC — phase3.yaml (synthesis gate, v0.6.x):** `0.7936` (95% CI: 0.6963–0.8827,
**Retrospective AUROC — pipeline.yaml (v0.7.x):** `0.8086` (95% CI: 0.7149–0.8957, n=2000
bootstrap). Positive-vs-negative separation on the 44-AMP + 44-background benchmark set using
the full ensemble scorer with pipeline.yaml weights. Historical progression: 0.7926 → 0.8138 →
0.8164 (PRs #48–54) → 0.8086 (PR #65: Trp-weighted aromatic bonus, safety abs() fix). The minor
decrease from 0.8164 reflects that some random-background decoys contain Trp residues and benefit
from the Trp bonus; this is within bootstrap CI width and the bonus is scientifically correct.

**Retrospective AUROC — phase3.yaml (synthesis gate, v0.7.x):** `0.7890` (95% CI: 0.6932–0.8784,
n=2000 bootstrap). Phase3 uses re-weighted ensemble scores (activity=0.35, safety=0.30,
synthesis=0.20, novelty=0.15 vs pipeline.yaml activity=0.40, safety=0.25, synthesis=0.15,
novelty=0.20) and a stricter safety gate (max_safety_risk=0.40). The AUROC is lower than
pipeline.yaml because the higher safety weight down-ranks some literature AMPs that have
hemolysis risk, which is scientifically correct behaviour for a synthesis gate. Interpretation:
**STRONG** (AUROC > 0.70). AUPRC = 0.8333 (+0.3333 above random baseline of 0.50).
Recall@10 = 0.23, Recall@20 = 0.43, Recall@44 = 0.68.
**STRONG** (AUROC > 0.70). AUPRC = 0.8301 (+0.3301 above random baseline of 0.50).
Recall@10 = 0.23, Recall@20 = 0.43, Recall@44 = 0.70.

> **Important:** The synthesis selection gate uses phase3.yaml. The phase3 AUROC (0.7936) is the
> operationally relevant benchmark. pipeline.yaml AUROC (0.8164) is the full-ensemble reference.
> **Important:** The synthesis selection gate uses phase3.yaml. The phase3 AUROC (0.7890) is the
> operationally relevant benchmark. pipeline.yaml AUROC (0.8086) is the full-ensemble reference.
> Both point estimates exceed the AUROC > 0.70 synthesis gate. Note that with n=88 sequences the
> 95% CI spans the gate boundary (0.6963–0.8827); synthesis decisions are made on point estimates,
> which is standard practice at this sample size — the CI reflects sampling uncertainty, not model
> unreliability.
> 95% CI spans the gate boundary; synthesis decisions are made on point estimates, which is standard
> practice at this sample size — the CI reflects sampling uncertainty, not model unreliability.

**AUPRC (v0.5.x):** `0.8556` for pipeline.yaml (+0.3556 above the random baseline of 0.50).
**AUPRC (v0.7.x):** `0.8495` for pipeline.yaml (+0.3495 above the random baseline of 0.50).
Area Under Precision-Recall Curve is computed alongside AUROC and reported by `make validate-scoring`.
AUPRC is preferable to AUROC for class-imbalanced datasets because it emphasises precision
at the operating point actually used for candidate selection. Random baseline = dataset
Expand All @@ -350,7 +350,7 @@ and avoid inflation to 1.0 for constant-score classifiers.
`make validate-scoring-phase3` (synthesis gate weights, phase3.yaml) or
`make validate-scoring-strict` for the composition-matched (scrambled-decoy) variant.

**Key scoring changes in v0.2.x–v0.5.x (PRs #48–58):**
**Key scoring changes in v0.2.x–v0.7.x (PRs #48–65):**

| PR | Change | AUROC impact |
|----|--------|-------------|
Expand All @@ -361,6 +361,7 @@ and avoid inflation to 1.0 for constant-score classifiers.
| #51 | Proline synthesis penalty (−0.10 at >15%); helix bonus 0.01→0.03 | 0.8138→0.8164 |
| #53 | Safety uses charge_density_ph74; stronger cytotox_penalty in pilot | pilot ranking |
| #58 | AUPRC added; SEED-009 (Bac2A) + SEED-010 (KR-12) added | +2 scaffolds |
| #65 | Trp-weighted aromatic bonus (1.5× vs Phe/Tyr); safety abs() fix | 0.8164→0.8086 |

**Limitation:** Benchmarks use a small, curated demo dataset (44 AMPs + 44 background). They
do not validate against large independent AMP databases. Real retrospective validation against
Expand Down
12 changes: 10 additions & 2 deletions src/openamp_foundry/features/physchem.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,10 @@
# Hydrophobic residues that drive aggregation in synthetic peptides: Val, Ile, Leu, Met, Phe, Trp
AGG_HYDROPHOBIC = set("VILMFW")

# Eisenberg consensus hydrophobicity scale (normalized, 0-centred removed, shifted to 0..1 range)
# Source: Eisenberg et al. (1984) J Mol Biol. Used for hydrophobic moment only.
# Eisenberg (1984) consensus hydrophobicity scale. Raw signed values: R=-2.530 (most hydrophilic)
# to I=+1.380 (most hydrophobic). DO NOT normalize to [0,1] — the hydrophobic moment calculation
# requires signed values so that polar/charged residues (negative) cancel apolar residues (positive).
# Source: Eisenberg et al. (1984) J Mol Biol 179:125-142, Table 2. Used for hydrophobic moment only.
_HYDROPHOBICITY: dict[str, float] = {
"A": 0.620, "R": -2.530, "N": -0.780, "D": -0.900, "C": 0.290,
"Q": -0.850, "E": -0.740, "G": 0.480, "H": -0.400, "I": 1.380,
Expand Down Expand Up @@ -71,6 +73,12 @@ def net_charge_at_ph74(sequence: str) -> float:

Reference: Bjellqvist et al. (1993) Electrophoresis 14:1023; pKa values for His
in peptides are shifted vs free amino acid (6.0) to 6.5 to account for context.

Note: α-amino N-terminus (+≈0.80 at pH 7.4, pKa ≈ 8.0) and α-carboxyl C-terminus
(−≈1.00 at pH 7.4, pKa ≈ 3.1) are excluded — only side-chain charges are computed.
The net terminal offset is ≈ −0.20 per peptide (~7% underestimation for a 10-mer).
This is the standard convention for relative charge comparisons across candidates
synthesized under the same conditions.
"""
charge = 0.0
for aa in sequence:
Expand Down
13 changes: 11 additions & 2 deletions src/openamp_foundry/scoring/activity.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,17 @@ def activity_likeness_score(features: dict) -> float:
# Hydrophobicity: 40-50% is a sweet spot for membrane interaction
hydro_score = 1.0 - min(abs(hydrophobic - 0.45) / 0.45, 1.0)

# Aromatic residues (F, W, Y) aid membrane insertion
aromatic_bonus = min(aromatic / 0.20, 1.0) * 0.10
# Aromatic residues (F, W, Y) aid membrane insertion.
# Trp is weighted 1.5× vs Phe/Tyr: Trp's indole ring preferentially anchors at the
# lipid-water interface (Wimley-White W=-1.85 kcal/mol; Wimley & White 1996 Nat Struct Biol),
# a mechanism distinct from and more membrane-favorable than Phe/Tyr's hydrophobic stacking.
# This differentiates puroindoline-a/indolicidin-class Trp-rich AMPs from Phe-rich peptides.
_counts = features.get("residue_counts", {})
_length = features["length"] or 1
trp_fraction = _counts.get("W", 0) / _length
non_trp_aromatic = max(0.0, aromatic - trp_fraction)
weighted_aromatic = trp_fraction * 1.5 + non_trp_aromatic
aromatic_bonus = min(weighted_aromatic / 0.20, 1.0) * 0.10

# Amphipathicity: helical hydrophobic moment > 0.4 is associated with activity
# Typical range for AMPs: 0.3–0.8; scale to [0,1] over 0–0.8 range
Expand Down
3 changes: 2 additions & 1 deletion src/openamp_foundry/scoring/boman.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
"""
from __future__ import annotations

import math

# Published amino acid interaction potentials from Boman (2003), Table 1, set 2.
# Units: kcal/mol (approximate interaction energies)
# Positive → hydrophilic/charged, negative → hydrophobic
Expand Down Expand Up @@ -86,7 +88,6 @@ def boman_activity_score(sequence: str) -> float:
bi = boman_index(sequence)
# Sigmoid-like normalization centered at 0, scaled so bi=1.0 → ≈ 0.73
# Using tanh mapping: score = 0.5 * (1 + tanh(bi / 2))
import math
score = 0.5 * (1.0 + math.tanh(bi / 2.0))
return round(max(0.0, min(1.0, score)), 4)

Expand Down
7 changes: 4 additions & 3 deletions src/openamp_foundry/scoring/safety.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,10 @@ def safety_score(features: dict) -> float:
typical AMP-like candidates (μH ≈ 0.35–0.50).
"""
length = features["length"]
# Prefer pH 7.4 charge (same key-preference as activity scorer); abs() applied because
# charge magnitude, not sign, drives hemolysis risk. Fall back for pre-PR-47 callers.
charge_density = abs(features.get("charge_density_ph74", features["charge_density"]))
# Prefer pH 7.4 charge density. Positive charge drives hemolysis (electrostatic membrane
# disruption); negatively charged peptides are repelled from both mammalian and bacterial
# membranes and pose no hemolysis risk. Do NOT use abs() here.
charge_density = features.get("charge_density_ph74", features["charge_density"])
hydrophobic = features["hydrophobic_fraction"]
cys = features["cysteine_fraction"]
repeat_run = features["longest_repeat_run"]
Expand Down
2 changes: 1 addition & 1 deletion tests/test_retrospective.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ def test_phase3_benchmark_passes_gate(self):
phase3.yaml shifts more weight to safety (0.30 vs 0.25) and synthesis (0.20 vs 0.15),
which down-ranks hemolytic AMPs — correct behaviour for a synthesis gate but lowers raw
AUROC slightly vs pipeline.yaml. Gate is still AUROC > 0.70 (STRONG).
Measured: AUROC=0.7936 (95% CI 0.6963-0.8827, n=2000 bootstrap).
Measured: AUROC=0.7890 (95% CI 0.6932-0.8784, n=2000 bootstrap; after PR #65 Trp bonus).

The upper-bound check (< 0.83) is a config-identity sentinel: pipeline.yaml scores
0.8164 which would breach it, so a silent config fallback is detectable.
Expand Down
Loading
Loading