diff --git a/docs/METHODS.md b/docs/METHODS.md index 4369f36d..25339653 100644 --- a/docs/METHODS.md +++ b/docs/METHODS.md @@ -320,6 +320,10 @@ Phase 2 benchmarks verified that the pipeline: benchmark set using the full ensemble scorer). Benchmarked after PRs #48–54; the scoring improvements in those PRs moved AUROC from 0.7926 → 0.8138 → 0.8164. +**To reproduce:** `make validate-scoring` (runs `cli validate-scoring` on +`examples/validation/known_amps.csv` vs `examples/validation/random_background.csv`) +or `make validate-scoring-strict` for the composition-matched (scrambled-decoy) variant. + **Key scoring changes in v0.2.x (PRs #48–54):** | PR | Change | AUROC impact | diff --git a/docs/RISK_REVIEW.md b/docs/RISK_REVIEW.md index e4250f30..a084d6ef 100644 --- a/docs/RISK_REVIEW.md +++ b/docs/RISK_REVIEW.md @@ -35,14 +35,18 @@ OpenAMP Foundry pipeline. It covers: | Risk | Status | Mitigation | |------|--------|-----------| -| Extreme hydrophobicity (hemolysis proxy) | **FILTERED** | max_safety_risk=0.40 in phase3.yaml excludes sequences with hydrophobic_fraction > 0.65 | -| Extreme charge density | **FILTERED** | Poly-cationic sequences excluded by safety scorer | +| Extreme hydrophobicity (hemolysis proxy) | **FILTERED** | max_safety_risk=0.40 in phase3.yaml excludes sequences with hydrophobic_fraction > 0.65 or strong amphipathicity (μH > 0.55) | +| Extreme charge density | **FILTERED** | Poly-cationic sequences excluded by safety scorer (charge_density_ph74 > 0.55 triggers risk increment); uses pH-7.4 charge for accuracy | +| Predicted mammalian cytotoxicity | **FILTERED + DOWN-RANKED** | Selectivity proxy (charge/GRAVY ratio) flags HIGH_CYTOTOX_RISK candidates (proxy < 0.5); cytotox_penalty doubly demotes them in pilot panel selection | | Long repeat runs (degenerate composition) | **FILTERED** | Sequences with repeat_run ≥ 6 penalised; few reach selection threshold | +| High proline content | **REPORTED** | proline_fraction > 15% incurs synthesis penalty (−0.10); flagged in expert review for pseudoproline dipeptide use | | Cysteines (disulfide bridges) | **REPORTED** | Flagged in synthesis feasibility report; reviewer must assess | | Sequences resembling known toxins | **NOT ASSESSED** | Computational toxin screening is out of scope for this pipeline version | **Action required:** A qualified reviewer must inspect the synthesis feasibility report for candidates with high cysteine fraction or proline content before ordering synthesis. +Candidates with HIGH_CYTOTOX_RISK flag (selectivity_proxy < 0.5) require extra hemolysis +assay scrutiny before any mammalian cell work. --- diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index f9b46925..9783750a 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,6 +1,6 @@ # Roadmap -## v0.1 — Safe starter +## v0.1 — Safe starter ✓ - toy data; - transparent heuristic scoring; @@ -8,40 +8,57 @@ - report generation; - tests and CI. -## v0.2 — Dataset layer +## v0.2 — Dataset layer ✓ - license-aware dataset loaders; - dataset cards; - deduplication; -- cluster splitting; -- leakage checks. +- cluster splitting (`benchmark/splits.py`); +- leakage checks (`bench leakage` CLI + `test_cluster_split.py`). -## v0.3 — Predictor adapters +## v0.3 — Predictor adapters ✓ -- adapters for open AMP predictors; -- versioned model manifests; -- ensemble scoring; -- uncertainty score. +- Boman index adapter (independent second scorer, `scoring/boman.py`); +- ensemble scoring with configurable weights (`scoring/ensemble.py`); +- model disagreement uncertainty signal (disagreement = |activity − boman|). -## v0.4 — Benchmark suite +## v0.4 — Benchmark suite ✓ -- hidden-positive recovery; -- negative control rejection; -- novelty stress test; -- toxicity penalty benchmark. +- hidden-positive recovery (`bench baseline`, `test_hidden_active_recovery.py`); +- negative control rejection (`test_negative_penalization.py`, `test_negative_robustness.py`); +- novelty stress test (`test_novelty_pressure.py`); +- toxicity penalty benchmark (`test_toxicity_penalty.py`); +- retrospective AUROC benchmark (`validate-scoring` CLI, AUROC = 0.8164 on demo set). -## v0.5 — Lab-batch package +## v0.5 — Lab-batch package ✓ -- pre-registration template; -- batch manifest; -- candidate certificates; -- expert-review checklist; -- safe publication template. +- pre-registration template (`docs/SELECTION_RULE.md`); +- batch manifest (`outputs/run_manifest.json` with SHA-256 input hashes); +- candidate certificates (JSON + `schemas/candidate.schema.json`); +- expert-review checklist (`docs/EXPERT_REVIEW_PACK.md`); +- wet-lab handoff guide (`docs/WET_LAB_HANDOFF.md`); +- safe publication template (`docs/NOMINATION_REPORT.md`). + +## v0.5.x — Scoring enhancements (shipped alongside v0.5) + +Implemented during the pre-wet-lab improvement loop (PRs #31–#54): + +- Serum stability scoring — HNE elastase + trypsin + chymotrypsin site density +- Selectivity proxy — charge/GRAVY-based mammalian cytotoxicity risk detector +- Pilot panel selection — `pilot_priority` formula with serum stability, novelty, selectivity, cytotox_penalty +- Aggregation propensity scoring — consecutive hydrophobic run + β-branched density +- Aggregation-safe mutation generation — prevents creating VILMFW runs ≥ 4 +- Proline synthesis penalty — DKP/coupling-difficulty penalty at > 15% proline +- Helix propensity bonus — Chou-Fasman Pα weight increased 0.01 → 0.03 in activity scorer +- Safety score uses pH-7.4 charge consistently with activity scorer +- C-terminal amidation and N-terminal acetylation recommendations in QC +- Wave 2 D-amino guidance in synthesis order +- DISCOVERY_PREDICTION.md — quantified discovery probability model (~29–49%) ## v1.0 — Validated dry-lab-to-wet-lab loop - independently reviewed assay batch; - lab results ingested; - negative results archived where safe; -- active-learning batch 2; +- active-learning batch 2 (D-amino variants, MDR strain panel); - public reproducibility report.