From c18519f9c9d174186c63c037f313fed2ba6ef8c6 Mon Sep 17 00:00:00 2001 From: j Date: Sun, 28 Jun 2026 16:37:19 +0700 Subject: [PATCH] docs: record AUPRC 0.8556 + CI 0.72-0.90 in benchmark section --- docs/DISCOVERY_PREDICTION.md | 3 ++- docs/METHODS.md | 11 +++++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/DISCOVERY_PREDICTION.md b/docs/DISCOVERY_PREDICTION.md index 2112c1fd..c3d606ae 100644 --- a/docs/DISCOVERY_PREDICTION.md +++ b/docs/DISCOVERY_PREDICTION.md @@ -243,7 +243,8 @@ Fmoc SPPS with acetonitrile/water gradient; verify MALDI-TOF pre-assay. Updated from ~55–65% due to addition of proven scaffold families. Basis: -- Pipeline AUROC = 0.8164 (bootstrap CI₉₅: 0.71–0.89) vs composition-matched UniProt decoys +- Pipeline AUROC = 0.8164 (95% CI: 0.72–0.90) vs composition-matched UniProt decoys +- Pipeline AUPRC = 0.8556 (+0.36 above random baseline 0.50; PR curve emphasises precision at selection operating point) - Recall@20 = 43% on internal benchmark (positives recovered in top 20 ranked candidates) - SEED-003 family (up to 4/20): RRWQWRMKKLG is curated known AMP → variants ~65–75% hit rate - SEED-007 family (up to 4/20): Bombolitin II (*Bombus pennsylvanicus* bumblebee venom) with diff --git a/docs/METHODS.md b/docs/METHODS.md index ecaea40e..1bcc6425 100644 --- a/docs/METHODS.md +++ b/docs/METHODS.md @@ -321,10 +321,13 @@ benchmark set using the full ensemble scorer). Benchmarked after PRs #48–54; t improvements in those PRs moved AUROC from 0.7926 → 0.8138 → 0.8164. Bootstrap 95% CI computed via Wilcoxon-Mann-Whitney (n=2000 iterations; CI reported in `validate-scoring` output). -**AUPRC (v0.5.x):** Area Under Precision-Recall Curve is now computed alongside AUROC and -reported by `make validate-scoring`. AUPRC is preferable to AUROC for class-imbalanced -datasets because it emphasises precision at the operating point actually used for candidate -selection. Random baseline = dataset prevalence (0.50 for the balanced demo set). +**AUPRC (v0.5.x):** `0.8556` (+0.3556 above the random baseline of 0.50). Area Under +Precision-Recall Curve is computed alongside AUROC and reported by `make validate-scoring`. +AUPRC is preferable to AUROC for class-imbalanced datasets because it emphasises precision +at the operating point actually used for candidate selection. Random baseline = dataset +prevalence (0.50 for the balanced demo set). Implemented with pessimistic tie-breaking +(negatives sort first on tied scores) to match sklearn `average_precision_score` convention +and avoid inflation to 1.0 for constant-score classifiers. **To reproduce:** `make validate-scoring` (runs `cli validate-scoring` on `examples/validation/known_amps.csv` vs `examples/validation/random_background.csv`)