Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,22 @@ All notable changes to OpenAMP Foundry are documented here.
**Status:** Pipeline frozen for synthesis batch ordering. All changes below were quality
improvements made before committing the ~$10k wet-lab synthesis budget.

### Latest fixes (PRs #61–69)

### Latest fixes (PRs #61–70)

- **PR #70** — Windowed hydrophobic moment (window=11, Eisenberg standard) + anionic charge guard:
`max_windowed_hydrophobic_moment()` added to `features/physchem.py`; `max_hydrophobic_moment`
feature now in `compute_features()` output; `activity_likeness_score()` uses
`max(hydrophobic_moment, max_hydrophobic_moment)` so long-sequence AMPs (magainin-2: 1.51×
improvement, cecropin-A: 2.90×) get credit for their best helical window rather than a
diluted full-sequence average; anionic guard returns 0.0 for peptides with
`charge_density_ph74 < 0.0` (bacterial membrane repulsion; corrects EEIEIEIEIEIEIEE
receiving 0.42 erroneously); fixed Python eager-evaluation bug in feature dict fallback
(`dict.get(key, features["k"])` → `if key in features` conditional); 23 new tests covering
windowed mu_h behaviour, max_hydrophobic_moment in compute_features(), anionic guard cases;
AUROC pipeline.yaml 0.8047→0.8348, phase3.yaml 0.7846→0.8126; regression sentinel updated
(≥0.79); config-identity sentinel updated (<0.82, sits between phase3=0.8126 and
pipeline=0.8348); METHODS.md and DISCOVERY_PREDICTION.md AUROC values updated;
1223 tests passing
- **PR #69** — synthesis_readiness_report.md updated for current 20-candidate pilot panel
(AUROC 0.8047; 43 AMPs; 6 seed families; all output files regenerated: presynth_qc_report.md,
diversity_report.md, gold_standard_calibration.md); external predictor gate marked PENDING for
Expand Down
10 changes: 5 additions & 5 deletions docs/DISCOVERY_PREDICTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
**Pipeline:** OpenAMP-Foundry v0.1.0
**Date:** 2026-06-28 (updated 2026-06-28)
**Status:** Pre-synthesis scientific assessment — for expert review before ordering
**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37), SEED-006 Mastoparan-X (PR #38), Charge×amphipathicity cross-term (PR #39), Amphipathicity weight + helix_bonus (PR #40), SEED-007 Bombolitin II + SEED-008 Puroindoline-a (PR #41), N-terminal acetylation flag + D-amino Wave 2 guidance (PR #42), Full synthesis risk QC — QG/QS deamidation, DG/DS isomerization, Trp photolability (PR #44), Vendor-ready synthesis order generator (PR #45), Diversity-aware pilot panel selection similarity-threshold=0.75 (PR #46), Selectivity proxy + HIGH_CYTOTOX_RISK flag — charge/GRAVY-based mammalian cytotoxicity risk detector (PR #47), selectivity_proxy routed into pilot_priority formula — low-cytotox-risk candidates gently demoted (PR #48), Elastase resistance (HNE 3-protease stability model) + aggregation propensity scoring (synthesis feasibility penalty) (PR #49), Aggregation-safe mutation generation + balanced K/R charge variants + SynthQC continuous aggregation score (PR #50), Proline synthesis penalty + helix bonus enhancement — AUROC 0.8138 → 0.8164 (PR #51), Safety pH74 charge consistency + stronger cytotoxicity penalty — SEED-004 net selectivity swing increased 0.035→0.055 (PR #53), AUPRC metric + SEED-009 (Bac2A proline-rich) + SEED-010 (KR-12 LL-37 fragment) (PR #58), ATCC strain corrections + preregistration assay protocol + negative result schema (PR #60), **CRITICAL FIX: SEED-008 Trp-rich scaffold reinstated in synthesis pool (was excluded by Boman W scale artifact); all 10 seeds generated; SEED-007/009 newly included; SEED-004 correctly excluded by safety gate; 6 mechanism-diverse scaffolds confirmed in synthesis pool** (PR #61), Phase3.yaml AUROC documented (0.7936→0.7890 after PR #65) (PR #63), Stale threshold/reference-count docs corrected (PR #64), **Trp-weighted aromatic bonus (1.5× Trp vs Phe/Tyr; Wimley-White interfacial mechanism); safety abs() bug fix (negative charge does not cause hemolysis); Eisenberg scale comment fix; pool SEED-006: 11→10, SEED-009: 18→19; AUROC 0.8164→0.8086** (PR #65), **Duplicate validation entry removed (REF-GIG-001 = magainin-2 = REF-MAG-001 counted twice); corrected AUROC 0.8086→0.8047 (pipeline), 0.7890→0.7846 (phase3); 43-AMP + 44-background set (n=87)** (PR #66)
**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37), SEED-006 Mastoparan-X (PR #38), Charge×amphipathicity cross-term (PR #39), Amphipathicity weight + helix_bonus (PR #40), SEED-007 Bombolitin II + SEED-008 Puroindoline-a (PR #41), N-terminal acetylation flag + D-amino Wave 2 guidance (PR #42), Full synthesis risk QC — QG/QS deamidation, DG/DS isomerization, Trp photolability (PR #44), Vendor-ready synthesis order generator (PR #45), Diversity-aware pilot panel selection similarity-threshold=0.75 (PR #46), Selectivity proxy + HIGH_CYTOTOX_RISK flag — charge/GRAVY-based mammalian cytotoxicity risk detector (PR #47), selectivity_proxy routed into pilot_priority formula — low-cytotox-risk candidates gently demoted (PR #48), Elastase resistance (HNE 3-protease stability model) + aggregation propensity scoring (synthesis feasibility penalty) (PR #49), Aggregation-safe mutation generation + balanced K/R charge variants + SynthQC continuous aggregation score (PR #50), Proline synthesis penalty + helix bonus enhancement — AUROC 0.8138 → 0.8164 (PR #51), Safety pH74 charge consistency + stronger cytotoxicity penalty — SEED-004 net selectivity swing increased 0.035→0.055 (PR #53), AUPRC metric + SEED-009 (Bac2A proline-rich) + SEED-010 (KR-12 LL-37 fragment) (PR #58), ATCC strain corrections + preregistration assay protocol + negative result schema (PR #60), **CRITICAL FIX: SEED-008 Trp-rich scaffold reinstated in synthesis pool (was excluded by Boman W scale artifact); all 10 seeds generated; SEED-007/009 newly included; SEED-004 correctly excluded by safety gate; 6 mechanism-diverse scaffolds confirmed in synthesis pool** (PR #61), Phase3.yaml AUROC documented (0.7936→0.7890 after PR #65) (PR #63), Stale threshold/reference-count docs corrected (PR #64), **Trp-weighted aromatic bonus (1.5× Trp vs Phe/Tyr; Wimley-White interfacial mechanism); safety abs() bug fix (negative charge does not cause hemolysis); Eisenberg scale comment fix; pool SEED-006: 11→10, SEED-009: 18→19; AUROC 0.8164→0.8086** (PR #65), **Duplicate validation entry removed (REF-GIG-001 = magainin-2 = REF-MAG-001 counted twice); corrected AUROC 0.8086→0.8047 (pipeline), 0.7890→0.7846 (phase3); 43-AMP + 44-background set (n=87)** (PR #66), **Windowed hydrophobic moment (window=11, Eisenberg standard) replaces full-sequence mu_h for activity scoring; anionic guard (charge_density_ph74 < 0.0 → score 0.0); Python eager-eval bug fix in feature dict fallback; 23 new tests; AUROC 0.8047→0.8348 (pipeline), 0.7846→0.8126 (phase3)** (PR #70)

---

Expand Down Expand Up @@ -243,7 +243,7 @@ Fmoc SPPS with acetonitrile/water gradient; verify MALDI-TOF pre-assay.
Updated from ~55–65% due to addition of proven scaffold families.

Basis:
- Pipeline AUROC = 0.8047 (95% CI: 0.71–0.89) vs composition-matched UniProt decoys (PR #66 dedup fix; 43 unique AMPs, n=87 total; previous 0.8086 pre-dedup, 0.8164 pre-Trp-bonus)
- Pipeline AUROC = 0.8348 (95% CI: TBD; PR #70 windowed mu_h + anionic guard; 43 unique AMPs, n=87 total; previous 0.8047 PR #66, 0.8086 pre-dedup, 0.8164 pre-Trp-bonus)
- Pipeline AUPRC = 0.8443 (+0.2943 above random baseline 0.4943; PR curve emphasises precision at selection operating point)
- Recall@20 = 43% on internal benchmark (positives recovered in top 20 ranked candidates)
- SEED-003 family (up to 4/20): RRWQWRMKKLG is curated known AMP → variants ~65–75% hit rate
Expand Down Expand Up @@ -607,7 +607,7 @@ Executing all four actions on the best Wave 1 hits would push the combined proba
| Stage | Gate | Original | After PRs #31–38 | After PRs #39–42 | After PRs #43–47 | After PRs #48–53 (current) | Primary limiting factor |
|-------|------|----------|-----------------|-----------------|-----------------|--------------------------|------------------------|
| 0 | Synthesis success | ~90% | ~90% | ~88% | **~89%** ✓ | **~90%** ✓ (agg model + agg-safe gen + pro penalty + pH74 charge) | SEED-008 W-rich; all aggregation/synthesis risks modelled |
| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~61–71%** ✓ (AUROC 0.8047) | AUROC 0.8047 (AUPRC 0.8443); 6 scaffold families confirmed |
| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~62–72%** ✓ (AUROC 0.8348) | AUROC 0.8348 (AUPRC 0.8443); 6 scaffold families confirmed |
| 2 | TI > 10 (selectivity) | ~35–50% | ~35–50% | ~38–52% | **~40–55%** ✓ | **~42–57%** ✓ (stronger SEED-004 demotion) | sel_proxy doubled penalty for HIGH_CYTOTOX_RISK tier |
| 3 | t½ > 2 h (serum) | ~10–20% | ~25–40% | ~28–42% | **~28–42%** | **~30–46%** ✓ (3-protease model + short/Trp-rich model correction; pilot-panel data) | SEED-003/008 may outperform model score; all seeds borderline — early serum assay recommended |
| 4 | Scaffold novelty | ~10–15% | ~18–25% | ~25–35% | **~26–36%** ✓ | **~26–36%** | Diversity filter removes cross-seed near-dups |
Expand Down Expand Up @@ -683,7 +683,7 @@ The corrected estimate is more conservative because:

### 2. Benchmark Limitations

The AUROC 0.8047 (phase3 synthesis gate: 0.7846) is measured on a small 43+44 demo dataset, not validated against the full
The AUROC 0.8348 (phase3 synthesis gate: 0.8126) is measured on a small 43+44 demo dataset, not validated against the full
APD3 (> 3,000 AMPs), DRAMP v3.0 (> 19,000 entries), or ESCAPE benchmark (> 80,000 peptides
from 27 repositories). This may overestimate discriminative power.

Expand Down Expand Up @@ -739,7 +739,7 @@ than the 10 originally planned. The path to 50%+ requires wet-lab data integrati
## Confidence Calibration

This assessment is based on:
- Internal benchmark AUROC = 0.8047 (n=87, bootstrap n=2000; phase3 synthesis gate = 0.7846; PR #66 dedup fix)
- Internal benchmark AUROC = 0.8348 (n=87, bootstrap n=2000; phase3 synthesis gate = 0.8126; PR #70 windowed mu_h + anionic guard)
- Literature hit rates for physchem AMP prediction (Loose et al. 2006; Tossi et al. 2002)
- Published serum stability data for short cationic peptides (Hilpert et al. 2006)
- D-amino acid t½ extension data (Wade et al. 1990, PNAS)
Expand Down
74 changes: 49 additions & 25 deletions docs/METHODS.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,20 +102,31 @@ seven terms; weights encode biochemical priors only (not data-optimised). Score
**Formula (Python source: `scoring/activity.py`):**

```
# Anionic guard (PR #70): if charge_density_ph74 < 0.0, return 0.0 immediately.
# Anionic peptides are electrostatically repelled by bacterial membranes.

μH_eff = max(hydrophobic_moment, max_hydrophobic_moment) # windowed mu_h (PR #70)

score = 0.24 × length_score
+ 0.27 × clamp01((charge_density_ph74 + 0.05) / 0.55)
+ 0.17 × (1 − min(|hydrophobic − 0.45| / 0.45, 1.0))
+ min(aromatic / 0.20, 1.0) × 0.10 # aromatic_bonus
+ clamp01(μH / 0.8) × 0.14 # amphipathicity_score
+ clamp01((Pα − 1.0) / 0.20) × 0.03 # helix_bonus
+ clamp01(charge_density_ph74 × μH / 0.15) × 0.02 # cross_bonus
+ min(aromatic / 0.20, 1.0) × 0.10 # aromatic_bonus
+ clamp01(μH_eff / 0.8) × 0.14 # amphipathicity_score (windowed)
+ clamp01((Pα − 1.0) / 0.20) × 0.03 # helix_bonus
+ clamp01(charge_density_ph74 × μH_eff / 0.15) × 0.02 # cross_bonus
# Maximum achievable: 0.24+0.27+0.17+0.10+0.14+0.03+0.02 = 0.97 (arithmetic ceiling;
# no explicit min() cap — the code returns clamp01(score) which clips to 1.0)
```

`μH_eff` = `max(hydrophobic_moment, max_hydrophobic_moment)`. `max_hydrophobic_moment` is the
best μH over any 11-residue window (Eisenberg standard). For sequences ≤ 11 AA this equals
the full-sequence μH. For longer sequences it captures the most amphipathic helical segment,
which better reflects the membrane-active region. (Added PR #70.)

`charge_density_ph74` is the net peptide charge at pH 7.4 (Henderson-Hasselbalch) divided
by sequence length; the code falls back to `charge_density` (integer net-charge proxy / length)
for callers that pre-date the pH-7.4 charge model.
for callers that pre-date the pH-7.4 charge model. Lookup uses explicit `if key in features`
conditional (not `dict.get()`) to avoid Python's eager evaluation of the fallback argument.

### 4.1b Boman Activity Score (Second Independent Scorer)

Expand Down Expand Up @@ -316,29 +327,26 @@ Phase 2 benchmarks verified that the pipeline:
- Produces stable rankings under repeated runs (reproducibility)
- Shows performance degradation when key scoring dimensions are ablated

**Retrospective AUROC — pipeline.yaml (v0.7.x):** `0.8047` (95% CI: 0.7072–0.8922, n=2000
bootstrap). Positive-vs-negative separation on the 43-AMP + 44-background benchmark set using
the full ensemble scorer with pipeline.yaml weights. Historical progression: 0.7926 → 0.8138 →
**Retrospective AUROC — pipeline.yaml (v0.8.x):** `0.8348` (95% CI: TBD, n=2000 bootstrap).
Positive-vs-negative separation on the 43-AMP + 44-background benchmark set using the full
ensemble scorer with pipeline.yaml weights. Historical progression: 0.7926 → 0.8138 →
0.8164 (PRs #48–54) → 0.8086 (PR #65: Trp-weighted aromatic bonus, safety abs() fix) →
0.8047 (PR #66: removed duplicate GIGKFLHSAKKFGKAFVGEIMNS / REF-GIG-001 from validation set;
43 unique AMPs used). The minor decrease from 0.8086 reflects that the removed entry (magainin-2)
is a high-ranking AMP; fewer true positives at the top slightly reduce AUROC. Scientifically
more accurate than the pre-dedup estimate.
0.8047 (PR #66: removed duplicate REF-GIG-001; 43 unique AMPs) →
0.8348 (PR #70: windowed mu_h + anionic guard; see below).

**Retrospective AUROC — phase3.yaml (synthesis gate, v0.7.x):** `0.7846` (95% CI: 0.6829–0.8747,
**Retrospective AUROC — phase3.yaml (synthesis gate, v0.8.x):** `0.8126` (95% CI: TBD,
n=2000 bootstrap). Phase3 uses re-weighted ensemble scores (activity=0.35, safety=0.30,
synthesis=0.20, novelty=0.15 vs pipeline.yaml activity=0.40, safety=0.25, synthesis=0.15,
novelty=0.20) and a stricter safety gate (max_safety_risk=0.40). The AUROC is lower than
pipeline.yaml because the higher safety weight down-ranks some literature AMPs that have
hemolysis risk, which is scientifically correct behaviour for a synthesis gate. Interpretation:
**STRONG** (AUROC > 0.70). AUPRC = 0.8240 (+0.3297 above random baseline of 0.4943).
Recall@10 = 0.23, Recall@20 = 0.44, Recall@43 = 0.70.

> **Important:** The synthesis selection gate uses phase3.yaml. The phase3 AUROC (0.7846) is the
> operationally relevant benchmark. pipeline.yaml AUROC (0.8047) is the full-ensemble reference.
> Both point estimates exceed the AUROC > 0.70 synthesis gate. Note that with n=87 sequences the
> 95% CI spans the gate boundary; synthesis decisions are made on point estimates, which is standard
> practice at this sample size — the CI reflects sampling uncertainty, not model unreliability.
novelty=0.20) and a stricter safety gate (max_safety_risk=0.40). Historical: 0.7846 (PR #66)
→ 0.8126 (PR #70). The AUROC is lower than pipeline.yaml because the higher safety weight
down-ranks some literature AMPs that have hemolysis risk, which is scientifically correct
behaviour for a synthesis gate. Interpretation: **STRONG** (AUROC > 0.70).

> **Important:** The synthesis selection gate uses phase3.yaml. The phase3 AUROC (0.8126) is the
> operationally relevant benchmark. pipeline.yaml AUROC (0.8348) is the full-ensemble reference.
> Both point estimates comfortably exceed the AUROC > 0.70 synthesis gate. Note that with n=87
> sequences the 95% CI spans ~±0.09; synthesis decisions are made on point estimates, which is
> standard practice at this sample size — the CI reflects sampling uncertainty, not model unreliability.

**AUPRC (v0.7.x):** `0.8443` for pipeline.yaml (+0.2943 above the random baseline of 0.4943).
Area Under Precision-Recall Curve is computed alongside AUROC and reported by `make validate-scoring`.
Expand All @@ -352,7 +360,7 @@ pessimistic tie-breaking (negatives sort first on tied scores) to match sklearn
`make validate-scoring-phase3` (synthesis gate weights, phase3.yaml) or
`make validate-scoring-strict` for the composition-matched (scrambled-decoy) variant.

**Key scoring changes in v0.2.x–v0.7.x (PRs #48–65):**
**Key scoring changes in v0.2.x–v0.8.x (PRs #48–70):**

| PR | Change | AUROC impact |
|----|--------|-------------|
Expand All @@ -365,6 +373,22 @@ pessimistic tie-breaking (negatives sort first on tied scores) to match sklearn
| #58 | AUPRC added; SEED-009 (Bac2A) + SEED-010 (KR-12) added | +2 scaffolds |
| #65 | Trp-weighted aromatic bonus (1.5× vs Phe/Tyr); safety abs() fix | 0.8164→0.8086 |
| #66 | Remove duplicate REF-GIG-001 (magainin-2 counted twice in validation set) | 0.8086→0.8047 |
| #70 | Windowed mu_h (window=11, Eisenberg standard); anionic charge guard | 0.8047→0.8348 (pipeline); 0.7846→0.8126 (phase3) |

**PR #70 detail — windowed hydrophobic moment:** For sequences > 11 residues, the full-sequence
μH is diluted by non-helical termini and linker regions. `max_windowed_hydrophobic_moment()`
computes the maximum μH over all 11-residue windows (the Eisenberg standard helical period).
For magainin-2 (23 AA): full-seq μH = 0.4548 vs windowed μH = 0.6862 (1.51×). For cecropin-A
(37 AA): full-seq μH = 0.2267 vs windowed μH = 0.6567 (2.90×). Short sequences (≤ 11 AA)
are unaffected: windowed == full-seq.

**PR #70 detail — anionic charge guard:** Anionic peptides (charge_density_ph74 < 0.0) are
electrostatically repelled by bacterial membranes (phosphatidylglycerol, cardiolipin surface)
and cannot operate via the canonical cationic AMP mechanism. The guard returns 0.0 immediately,
preventing partial credit from hydrophobic/length terms. Boundary: exactly 0.0 charge density
is not blocked (reflects neutral peptides rather than anionic). Fixed a Python eagerness bug
in feature dict lookup (`features.get(key, features["fallback"])` evaluates fallback eagerly;
replaced with explicit `if "key" in features` conditional).

**Limitation:** Benchmarks use a small, curated demo dataset (43 AMPs + 44 background). They
do not validate against large independent AMP databases. Real retrospective validation against
Expand Down
Loading
Loading