Repository navigation
docs: wet-lab probability analysis, gold standard calibration, AUROC fix - #81
Merged
Merged
Conversation
…x AUROC - docs/WET_LAB_PROBABILITY.md: quantified per-family and composite discovery probability for 20-candidate pilot panel across 7 scaffold families; includes attrition model, breaking-news criteria, assay sequence guidance, and documented epistemic ceiling on computational prediction - docs/GOLD_STANDARD_CALIBRATION.md: committed calibration report showing known AMPs (Magainin-2, Melittin, Defensin-HNP1, Temporin-A) score within/above the current 7-family pilot panel (mean ensemble 0.821); confirms pipeline calibration - cli.py: fix AUROC=0.8047→0.8420 in gold_standard disclaimer text (AUROC 0.8047 was the pre-PR #66 duplicate-corrected value; 0.8420 is the PR #72 final value)
GOLD_STANDARD_CALIBRATION.md: - [HIGH] Fix Melittin blind-spot text: safety model assigns Safety=1.0 for Melittin (a confirmed model failure for curvature-mediated hemolysis); warn that Safety=1.0 cannot be used as evidence of non-hemolysis for any panel candidate - [MEDIUM] Acknowledge LL-37/Cecropin-A scoring below panel floor — indicates pipeline is optimised for magainin/mastoparan class; LL-37-like mechanisms may be under-scored - [MEDIUM] Rephrase AUROC disclaimer: "retrospective benchmark, not external prospective validation" (avoids conflating benchmark performance with independence) - [LOW] Fix Temporin-A SEED-004 reference for readers without history context WET_LAB_PROBABILITY.md: - [HIGH] Fix arithmetic error in pessimistic composite: 0.000127 → 0.000635, "99.99%" → "99.94%" - [MEDIUM] Fix SEED-009 mechanism description: ribosome exit tunnel binding (Gagnon 2016) and DnaK chaperone binding are distinct intracellular targets, not one pathway - [LOW] Fix "lowest serum stability" text: SEED-003 and SEED-008 (not SEED-003 and SEED-005) have the lowest model-estimated stability scores - Downgrade hemolysis probability estimates in conditional table: safety model is blind to curvature-mediated hemolysis (Melittin confirmed); estimate from structural class literature instead; adjust P(active AND passes hemolysis) 55-70% → 45-60%
This was referenced Jun 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/WET_LAB_PROBABILITY.md— Comprehensive per-family and composite probability analysis for the 20-candidate pilot panel across 7 scaffold families. Covers: P(MIC ≤ 16 µg/mL) = 92–97%, P(potent, ≤4 µg/mL) = 65–80%, P(publishable) = 30–50%, P(breaking news with MDR) = 8–18%. Includes attrition model, assay sequence guidance, and documented epistemic ceiling (why 100% probability is computationally impossible).docs/GOLD_STANDARD_CALIBRATION.md— Committed calibration report (previously only in gitignoredoutputs/). Magainin-2 scores ABOVE the panel (0.872); Melittin, Defensin-HNP1, Temporin-A score WITHIN; LL-37/Cecropin-A score BELOW. Confirms internal consistency at AUROC 0.8420.cli.pyAUROC fix —gold_standarddisclaimer text had staleAUROC=0.8047(pre-PR fix: remove duplicate validation entry (REF-GIG-001 = magainin-2) #66 value); corrected toAUROC=0.8420(PR feat: face segregation bonus in activity scoring (+AUROC 0.8348→0.8420) #72 final).Stale output files (gitignored, regenerated locally)
outputs/pilot_panel.fastaandoutputs/external_predict_checklist.mdwere stale (pre-PR #72 panel, missing SEED-001_VAR_064, wrong SEED-008 variant). Regenerated by runningmake external-predictwhich re-ran the full pipeline. Both files now reflect the current 20-candidate 7-family panel. (These files are gitignored per.gitignore; they can be regenerated at any time withmake external-predict.)Test plan
make test)pilot_panel.fastacontains SEED-001_VAR_064 (rank 20) and correct 7-family 20-candidate panelgrep "AUROC=0.8420" src/openamp_foundry/cli.pypasses🤖 Generated with Claude Code