Repository navigation
fix: remove duplicate validation entry (REF-GIG-001 = magainin-2) - #66
Merged
Merged
Conversation
…ted twice) GIGKFLHSAKKFGKAFVGEIMNS appeared as both REF-MAG-001 and REF-GIG-001 in examples/validation/known_amps.csv, inflating the AUROC benchmark by double-counting one high-ranking positive. Removed REF-GIG-001 (mislabeled "template_seed_2"). Corrected benchmark metrics (43 AMPs + 44 background, n=87): - pipeline.yaml: AUROC 0.8086 → 0.8047 (95% CI 0.7072–0.8922, AUPRC 0.8443) - phase3.yaml: AUROC 0.7890 → 0.7846 (95% CI 0.6829–0.8747, AUPRC 0.8240) Both configs remain STRONG (> 0.70) and phase3 stays above the 0.75 regression sentinel. Updated test_retrospective.py docstring and sentinel comments. Updated METHODS.md scoring history table (PR #66 row added) and DISCOVERY_PREDICTION.md AUROC references.
HIGH: config-identity sentinel threshold 0.83→0.795 (pipeline AUROC=0.8047 < 0.83 meant both configs passed; 0.795 sits between phase3=0.7846 and pipeline=0.8047) MEDIUM fixes: - METHODS.md: phase3 AUPRC baseline 0.50→0.4943; delta 0.3240→0.3297 (post-dedup prevalence) - DISCOVERY_PREDICTION.md: stage gate table still had stale (AUROC 0.8086) in one cell - cli.py: stale AUROC=0.8037/44 AMPs→0.8047/43 AMPs in user-facing disclaimer text - scrambled_decoys.csv: removed orphaned DECOY-GIG-001 (paired with deleted REF-GIG-001); strict benchmark restored to 43+43 balanced design LOW fixes: - retrospective.py: module docstring 44→43 AMPs in both modes; default recall_ks [10,20,44]→[10,20,43] - cli.py output key: recall_at_44→recall_at_43 to match updated default - test_benchmark_evaluate.py: comment/inputs updated from 44/88→43/87 All 1191 tests pass.
1 of 3 tasks
cschanhniem
added a commit
that referenced
this pull request
Jun 28, 2026
AUROC 0.8420 was achieved in PR #72 (face_segregation_bonus), not in the initial 5-seed benchmark generation run. Clarify that the generation section describes the initial corpus and the AUROC figure reflects scoring improvements through PR #72. Also note that the 87-sequence eval set was finalized in PR #66.
cschanhniem
added a commit
that referenced
this pull request
Jun 28, 2026
* docs: update METHODS.md version tag and annotate 7-seed synthesis pool expansion The methods document was last updated for PRs #47-54. Since then the synthesis pool was expanded from 5 to 7 scaffold families (SEED-006 through SEED-009 in expansion PRs; SEED-001 re-entered in PR #72 via face_segregation_bonus). Changes: - Version line: "PRs #47-54" → "PRs #47-78" - Generation section: added blockquote note distinguishing the initial 5-seed benchmark run (383 variants, 87-candidate eval set, AUROC 0.8420) from the current 7-seed synthesis pool (100 candidates) - No scoring formulas or benchmark numbers changed * docs: fix AUROC attribution in METHODS.md 7-seed note AUROC 0.8420 was achieved in PR #72 (face_segregation_bonus), not in the initial 5-seed benchmark generation run. Clarify that the generation section describes the initial corpus and the AUROC figure reflects scoring improvements through PR #72. Also note that the 87-sequence eval set was finalized in PR #66.
4 of 5 tasks
cschanhniem
added a commit
that referenced
this pull request
Jun 28, 2026
…fix (#81) * docs: add wet-lab probability analysis, gold standard calibration, fix AUROC - docs/WET_LAB_PROBABILITY.md: quantified per-family and composite discovery probability for 20-candidate pilot panel across 7 scaffold families; includes attrition model, breaking-news criteria, assay sequence guidance, and documented epistemic ceiling on computational prediction - docs/GOLD_STANDARD_CALIBRATION.md: committed calibration report showing known AMPs (Magainin-2, Melittin, Defensin-HNP1, Temporin-A) score within/above the current 7-family pilot panel (mean ensemble 0.821); confirms pipeline calibration - cli.py: fix AUROC=0.8047→0.8420 in gold_standard disclaimer text (AUROC 0.8047 was the pre-PR #66 duplicate-corrected value; 0.8420 is the PR #72 final value) * fix: address code review findings on calibration and probability docs GOLD_STANDARD_CALIBRATION.md: - [HIGH] Fix Melittin blind-spot text: safety model assigns Safety=1.0 for Melittin (a confirmed model failure for curvature-mediated hemolysis); warn that Safety=1.0 cannot be used as evidence of non-hemolysis for any panel candidate - [MEDIUM] Acknowledge LL-37/Cecropin-A scoring below panel floor — indicates pipeline is optimised for magainin/mastoparan class; LL-37-like mechanisms may be under-scored - [MEDIUM] Rephrase AUROC disclaimer: "retrospective benchmark, not external prospective validation" (avoids conflating benchmark performance with independence) - [LOW] Fix Temporin-A SEED-004 reference for readers without history context WET_LAB_PROBABILITY.md: - [HIGH] Fix arithmetic error in pessimistic composite: 0.000127 → 0.000635, "99.99%" → "99.94%" - [MEDIUM] Fix SEED-009 mechanism description: ribosome exit tunnel binding (Gagnon 2016) and DnaK chaperone binding are distinct intracellular targets, not one pathway - [LOW] Fix "lowest serum stability" text: SEED-003 and SEED-008 (not SEED-003 and SEED-005) have the lowest model-estimated stability scores - Downgrade hemolysis probability estimates in conditional table: safety model is blind to curvature-mediated hemolysis (Melittin confirmed); estimate from structural class literature instead; adjust P(active AND passes hemolysis) 55-70% → 45-60%
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GIGKFLHSAKKFGKAFVGEIMNS(magainin-2) appeared as bothREF-MAG-001andREF-GIG-001inexamples/validation/known_amps.csv, inflating the AUROC benchmark by double-counting one high-ranking positiveREF-GIG-001(mislabeled astemplate_seed_2— the same sequence asSEED-002inamp_seeds.csv)Corrected benchmark metrics
Both configs remain STRONG (AUROC > 0.70). The phase3 regression sentinel (≥ 0.75) still passes at 0.7846. The config-identity sentinel (< 0.83) still correctly flags any pipeline.yaml / phase3.yaml confusion.
Files changed
examples/validation/known_amps.csv— removed duplicate row (REF-GIG-001)tests/test_retrospective.py— updated docstring and sentinel comments to post-dedup valuesdocs/METHODS.md— updated AUROC/AUPRC/CI values, benchmark set size (43+44), added PR fix: remove duplicate validation entry (REF-GIG-001 = magainin-2) #66 to scoring history tabledocs/DISCOVERY_PREDICTION.md— updated all AUROC/AUPRC references to post-dedup valuesTest plan
pytest tests/ -q→ 1191 passedmake validate-scoring→ AUROC 0.8047 STRONGmake validate-scoring-phase3→ AUROC 0.7846 STRONG, ≥0.75 sentinel passes🤖 Generated with Claude Code