diff --git a/docs/DISCOVERY_PREDICTION.md b/docs/DISCOVERY_PREDICTION.md index 5a23a839..7ac485a3 100644 --- a/docs/DISCOVERY_PREDICTION.md +++ b/docs/DISCOVERY_PREDICTION.md @@ -3,7 +3,7 @@ **Pipeline:** OpenAMP-Foundry v0.1.0 **Date:** 2026-06-28 (updated 2026-06-28) **Status:** Pre-synthesis scientific assessment — for expert review before ordering -**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37), SEED-006 Mastoparan-X (PR #38), Charge×amphipathicity cross-term (PR #39), Amphipathicity weight + helix_bonus (PR #40), SEED-007 Bombolitin II + SEED-008 Puroindoline-a (PR #41), N-terminal acetylation flag + D-amino Wave 2 guidance (PR #42), Full synthesis risk QC — QG/QS deamidation, DG/DS isomerization, Trp photolability (PR #44), Vendor-ready synthesis order generator (PR #45), Diversity-aware pilot panel selection similarity-threshold=0.75 (PR #46), Selectivity proxy + HIGH_CYTOTOX_RISK flag — charge/GRAVY-based mammalian cytotoxicity risk detector (PR #47), selectivity_proxy routed into pilot_priority formula — low-cytotox-risk candidates gently demoted (PR #48), Elastase resistance (HNE 3-protease stability model) + aggregation propensity scoring (synthesis feasibility penalty) (PR #49), Aggregation-safe mutation generation + balanced K/R charge variants + SynthQC continuous aggregation score (PR #50), Proline synthesis penalty + helix bonus enhancement — AUROC 0.8138 → 0.8164 (PR #51), Safety pH74 charge consistency + stronger cytotoxicity penalty — SEED-004 net selectivity swing increased 0.035→0.055 (PR #53), AUPRC metric + SEED-009 (Bac2A proline-rich) + SEED-010 (KR-12 LL-37 fragment) (PR #58), ATCC strain corrections + preregistration assay protocol + negative result schema (PR #60), **CRITICAL FIX: SEED-008 Trp-rich scaffold reinstated in synthesis pool (was excluded by Boman W scale artifact); all 10 seeds generated; SEED-007/009 newly included; SEED-004 correctly excluded by safety gate; 6 mechanism-diverse scaffolds confirmed in synthesis pool** (PR #61), Phase3.yaml AUROC documented (0.7936→0.7890 after PR #65) (PR #63), Stale threshold/reference-count docs corrected (PR #64), **Trp-weighted aromatic bonus (1.5× Trp vs Phe/Tyr; Wimley-White interfacial mechanism); safety abs() bug fix (negative charge does not cause hemolysis); Eisenberg scale comment fix; pool SEED-006: 11→10, SEED-009: 18→19; AUROC 0.8164→0.8086** (PR #65) +**Completed improvements:** Serum stability scoring (PR #31/#32), Family diversity cap (PR #31), Reference set expansion 44→73 sequences (PR #33), Net charge pH 7.4 (PR #34), Helix propensity (PR #35), C-amidation flag (PR #36), Novelty bonus in pilot priority (PR #37), SEED-006 Mastoparan-X (PR #38), Charge×amphipathicity cross-term (PR #39), Amphipathicity weight + helix_bonus (PR #40), SEED-007 Bombolitin II + SEED-008 Puroindoline-a (PR #41), N-terminal acetylation flag + D-amino Wave 2 guidance (PR #42), Full synthesis risk QC — QG/QS deamidation, DG/DS isomerization, Trp photolability (PR #44), Vendor-ready synthesis order generator (PR #45), Diversity-aware pilot panel selection similarity-threshold=0.75 (PR #46), Selectivity proxy + HIGH_CYTOTOX_RISK flag — charge/GRAVY-based mammalian cytotoxicity risk detector (PR #47), selectivity_proxy routed into pilot_priority formula — low-cytotox-risk candidates gently demoted (PR #48), Elastase resistance (HNE 3-protease stability model) + aggregation propensity scoring (synthesis feasibility penalty) (PR #49), Aggregation-safe mutation generation + balanced K/R charge variants + SynthQC continuous aggregation score (PR #50), Proline synthesis penalty + helix bonus enhancement — AUROC 0.8138 → 0.8164 (PR #51), Safety pH74 charge consistency + stronger cytotoxicity penalty — SEED-004 net selectivity swing increased 0.035→0.055 (PR #53), AUPRC metric + SEED-009 (Bac2A proline-rich) + SEED-010 (KR-12 LL-37 fragment) (PR #58), ATCC strain corrections + preregistration assay protocol + negative result schema (PR #60), **CRITICAL FIX: SEED-008 Trp-rich scaffold reinstated in synthesis pool (was excluded by Boman W scale artifact); all 10 seeds generated; SEED-007/009 newly included; SEED-004 correctly excluded by safety gate; 6 mechanism-diverse scaffolds confirmed in synthesis pool** (PR #61), Phase3.yaml AUROC documented (0.7936→0.7890 after PR #65) (PR #63), Stale threshold/reference-count docs corrected (PR #64), **Trp-weighted aromatic bonus (1.5× Trp vs Phe/Tyr; Wimley-White interfacial mechanism); safety abs() bug fix (negative charge does not cause hemolysis); Eisenberg scale comment fix; pool SEED-006: 11→10, SEED-009: 18→19; AUROC 0.8164→0.8086** (PR #65), **Duplicate validation entry removed (REF-GIG-001 = magainin-2 = REF-MAG-001 counted twice); corrected AUROC 0.8086→0.8047 (pipeline), 0.7890→0.7846 (phase3); 43-AMP + 44-background set (n=87)** (PR #66) --- @@ -243,8 +243,8 @@ Fmoc SPPS with acetonitrile/water gradient; verify MALDI-TOF pre-assay. Updated from ~55–65% due to addition of proven scaffold families. Basis: -- Pipeline AUROC = 0.8086 (95% CI: 0.71–0.90) vs composition-matched UniProt decoys (PR #65 update; previous 0.8164 after Trp bonus scientific fix; within bootstrap CI) -- Pipeline AUPRC = 0.8495 (+0.35 above random baseline 0.50; PR curve emphasises precision at selection operating point) +- Pipeline AUROC = 0.8047 (95% CI: 0.71–0.89) vs composition-matched UniProt decoys (PR #66 dedup fix; 43 unique AMPs, n=87 total; previous 0.8086 pre-dedup, 0.8164 pre-Trp-bonus) +- Pipeline AUPRC = 0.8443 (+0.2943 above random baseline 0.4943; PR curve emphasises precision at selection operating point) - Recall@20 = 43% on internal benchmark (positives recovered in top 20 ranked candidates) - SEED-003 family (up to 4/20): RRWQWRMKKLG is curated known AMP → variants ~65–75% hit rate - SEED-007 family (up to 4/20): Bombolitin II (*Bombus pennsylvanicus* bumblebee venom) with @@ -584,7 +584,7 @@ Executing all four actions on the best Wave 1 hits would push the combined proba | Stage | Gate | Original | After PRs #31–38 | After PRs #39–42 | After PRs #43–47 | After PRs #48–53 (current) | Primary limiting factor | |-------|------|----------|-----------------|-----------------|-----------------|--------------------------|------------------------| | 0 | Synthesis success | ~90% | ~90% | ~88% | **~89%** ✓ | **~90%** ✓ (agg model + agg-safe gen + pro penalty + pH74 charge) | SEED-008 W-rich; all aggregation/synthesis risks modelled | -| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~61–71%** ✓ (AUROC 0.8086) | AUROC 0.8086 (AUPRC 0.8495); 6 scaffold families confirmed | +| 1 | MIC ≤ 32 μg/mL | ~55–65% | ~55–65% | ~60–70% | **~60–70%** | **~61–71%** ✓ (AUROC 0.8047) | AUROC 0.8047 (AUPRC 0.8443); 6 scaffold families confirmed | | 2 | TI > 10 (selectivity) | ~35–50% | ~35–50% | ~38–52% | **~40–55%** ✓ | **~42–57%** ✓ (stronger SEED-004 demotion) | sel_proxy doubled penalty for HIGH_CYTOTOX_RISK tier | | 3 | t½ > 2 h (serum) | ~10–20% | ~25–40% | ~28–42% | **~28–42%** | **~29–44%** ✓ (3-protease model) | Wave 2 D-amino plan machine-readable | | 4 | Scaffold novelty | ~10–15% | ~18–25% | ~25–35% | **~26–36%** ✓ | **~26–36%** | Diversity filter removes cross-seed near-dups | @@ -660,7 +660,7 @@ The corrected estimate is more conservative because: ### 2. Benchmark Limitations -The AUROC 0.8086 (phase3 synthesis gate: 0.7890) is measured on a small 44+44 demo dataset, not validated against the full +The AUROC 0.8047 (phase3 synthesis gate: 0.7846) is measured on a small 43+44 demo dataset, not validated against the full APD3 (> 3,000 AMPs), DRAMP v3.0 (> 19,000 entries), or ESCAPE benchmark (> 80,000 peptides from 27 repositories). This may overestimate discriminative power. @@ -716,7 +716,7 @@ than the 10 originally planned. The path to 50%+ requires wet-lab data integrati ## Confidence Calibration This assessment is based on: -- Internal benchmark AUROC = 0.8086 (n=88, bootstrap n=2000; phase3 synthesis gate = 0.7890; PR #65 Trp-weighted bonus update) +- Internal benchmark AUROC = 0.8047 (n=87, bootstrap n=2000; phase3 synthesis gate = 0.7846; PR #66 dedup fix) - Literature hit rates for physchem AMP prediction (Loose et al. 2006; Tossi et al. 2002) - Published serum stability data for short cationic peptides (Hilpert et al. 2006) - D-amino acid t½ extension data (Wade et al. 1990, PNAS) diff --git a/docs/METHODS.md b/docs/METHODS.md index 8cf735f9..7feccf3b 100644 --- a/docs/METHODS.md +++ b/docs/METHODS.md @@ -316,35 +316,37 @@ Phase 2 benchmarks verified that the pipeline: - Produces stable rankings under repeated runs (reproducibility) - Shows performance degradation when key scoring dimensions are ablated -**Retrospective AUROC — pipeline.yaml (v0.7.x):** `0.8086` (95% CI: 0.7149–0.8957, n=2000 -bootstrap). Positive-vs-negative separation on the 44-AMP + 44-background benchmark set using +**Retrospective AUROC — pipeline.yaml (v0.7.x):** `0.8047` (95% CI: 0.7072–0.8922, n=2000 +bootstrap). Positive-vs-negative separation on the 43-AMP + 44-background benchmark set using the full ensemble scorer with pipeline.yaml weights. Historical progression: 0.7926 → 0.8138 → -0.8164 (PRs #48–54) → 0.8086 (PR #65: Trp-weighted aromatic bonus, safety abs() fix). The minor -decrease from 0.8164 reflects that some random-background decoys contain Trp residues and benefit -from the Trp bonus; this is within bootstrap CI width and the bonus is scientifically correct. +0.8164 (PRs #48–54) → 0.8086 (PR #65: Trp-weighted aromatic bonus, safety abs() fix) → +0.8047 (PR #66: removed duplicate GIGKFLHSAKKFGKAFVGEIMNS / REF-GIG-001 from validation set; +43 unique AMPs used). The minor decrease from 0.8086 reflects that the removed entry (magainin-2) +is a high-ranking AMP; fewer true positives at the top slightly reduce AUROC. Scientifically +more accurate than the pre-dedup estimate. -**Retrospective AUROC — phase3.yaml (synthesis gate, v0.7.x):** `0.7890` (95% CI: 0.6932–0.8784, +**Retrospective AUROC — phase3.yaml (synthesis gate, v0.7.x):** `0.7846` (95% CI: 0.6829–0.8747, n=2000 bootstrap). Phase3 uses re-weighted ensemble scores (activity=0.35, safety=0.30, synthesis=0.20, novelty=0.15 vs pipeline.yaml activity=0.40, safety=0.25, synthesis=0.15, novelty=0.20) and a stricter safety gate (max_safety_risk=0.40). The AUROC is lower than pipeline.yaml because the higher safety weight down-ranks some literature AMPs that have hemolysis risk, which is scientifically correct behaviour for a synthesis gate. Interpretation: -**STRONG** (AUROC > 0.70). AUPRC = 0.8301 (+0.3301 above random baseline of 0.50). -Recall@10 = 0.23, Recall@20 = 0.43, Recall@44 = 0.70. +**STRONG** (AUROC > 0.70). AUPRC = 0.8240 (+0.3297 above random baseline of 0.4943). +Recall@10 = 0.23, Recall@20 = 0.44, Recall@43 = 0.70. -> **Important:** The synthesis selection gate uses phase3.yaml. The phase3 AUROC (0.7890) is the -> operationally relevant benchmark. pipeline.yaml AUROC (0.8086) is the full-ensemble reference. -> Both point estimates exceed the AUROC > 0.70 synthesis gate. Note that with n=88 sequences the +> **Important:** The synthesis selection gate uses phase3.yaml. The phase3 AUROC (0.7846) is the +> operationally relevant benchmark. pipeline.yaml AUROC (0.8047) is the full-ensemble reference. +> Both point estimates exceed the AUROC > 0.70 synthesis gate. Note that with n=87 sequences the > 95% CI spans the gate boundary; synthesis decisions are made on point estimates, which is standard > practice at this sample size — the CI reflects sampling uncertainty, not model unreliability. -**AUPRC (v0.7.x):** `0.8495` for pipeline.yaml (+0.3495 above the random baseline of 0.50). +**AUPRC (v0.7.x):** `0.8443` for pipeline.yaml (+0.2943 above the random baseline of 0.4943). Area Under Precision-Recall Curve is computed alongside AUROC and reported by `make validate-scoring`. AUPRC is preferable to AUROC for class-imbalanced datasets because it emphasises precision at the operating point actually used for candidate selection. Random baseline = dataset -prevalence (0.50 for the balanced demo set). Implemented with pessimistic tie-breaking -(negatives sort first on tied scores) to match sklearn `average_precision_score` convention -and avoid inflation to 1.0 for constant-score classifiers. +prevalence (43/87 ≈ 0.4943 for the slightly unbalanced post-dedup set). Implemented with +pessimistic tie-breaking (negatives sort first on tied scores) to match sklearn +`average_precision_score` convention and avoid inflation to 1.0 for constant-score classifiers. **To reproduce:** `make validate-scoring` (pipeline.yaml weights) or `make validate-scoring-phase3` (synthesis gate weights, phase3.yaml) or @@ -362,8 +364,9 @@ and avoid inflation to 1.0 for constant-score classifiers. | #53 | Safety uses charge_density_ph74; stronger cytotox_penalty in pilot | pilot ranking | | #58 | AUPRC added; SEED-009 (Bac2A) + SEED-010 (KR-12) added | +2 scaffolds | | #65 | Trp-weighted aromatic bonus (1.5× vs Phe/Tyr); safety abs() fix | 0.8164→0.8086 | +| #66 | Remove duplicate REF-GIG-001 (magainin-2 counted twice in validation set) | 0.8086→0.8047 | -**Limitation:** Benchmarks use a small, curated demo dataset (44 AMPs + 44 background). They +**Limitation:** Benchmarks use a small, curated demo dataset (43 AMPs + 44 background). They do not validate against large independent AMP databases. Real retrospective validation against APD3-scale data (> 3,000 AMPs, cluster-split) is required before strong claims of predictive power. diff --git a/examples/validation/known_amps.csv b/examples/validation/known_amps.csv index 73e9ce13..2b1a6123 100644 --- a/examples/validation/known_amps.csv +++ b/examples/validation/known_amps.csv @@ -24,7 +24,6 @@ REF-CEC-001,KWKLFKKIEKVGQNIRDGIIKAGPAV,cecropin_a_fragment,Steiner_1981_PNAS,1 REF-CEC-002,KWKIFKKIEKVGRNVRDGIIKAGP,cecropin_b_fragment,Hultmark_1980_Eur_J_Biochem,1 REF-MEL-001,GIGAVLKVLTTGLPALISWIKRKRQQ,melittin,Habermann_1972,1 REF-KWK-001,KWKLFKKIGAVLKVL,template_seed_1,Oren_1997_Biochemistry,1 -REF-GIG-001,GIGKFLHSAKKFGKAFVGEIMNS,template_seed_2,Zasloff_1987_PNAS,1 REF-MAG-004,KKLKKFGLRIINKIVAAL,magainin_helix,Wieprecht_1997,1 REF-BMA-001,GRFKRFRKKFKKLFKKLSP,bmap_fragment,Skerlavaj_1999,1 REF-LLL-001,LLRWPWWPWRRK,omiganan_analog,Sader_2004,1 diff --git a/examples/validation/scrambled_decoys.csv b/examples/validation/scrambled_decoys.csv index 14473ae1..5a106404 100644 --- a/examples/validation/scrambled_decoys.csv +++ b/examples/validation/scrambled_decoys.csv @@ -24,7 +24,6 @@ DECOY-CEC-001,KRWNVKKIIQKGAPDIKIGEFLVKGA,shuffled_cecropin_a_fragment,REF-CEC-00 DECOY-CEC-002,KAGIKDIRIRIKGGWPNVKEKFVK,shuffled_cecropin_b_fragment,REF-CEC-002,0 DECOY-MEL-001,ILTLIKTLIGQALGWKVPSQARVRKG,shuffled_melittin,REF-MEL-001,0 DECOY-KWK-001,FALKGWKKLVLIKVK,shuffled_template_seed_1,REF-KWK-001,0 -DECOY-GIG-001,KKGHASFIFIGLEKNSVFMGAKG,shuffled_template_seed_2,REF-GIG-001,0 DECOY-MAG-004,ALLKLAFKKKVGKRIIIN,shuffled_magainin_helix,REF-MAG-004,0 DECOY-BMA-001,KKGRKPLFFRLFKRKKSKF,shuffled_bmap_fragment,REF-BMA-001,0 DECOY-LLL-001,WLWPRLWKPRWR,shuffled_omiganan_analog,REF-LLL-001,0 diff --git a/src/openamp_foundry/benchmark/retrospective.py b/src/openamp_foundry/benchmark/retrospective.py index 3fd3465b..b9bee27f 100644 --- a/src/openamp_foundry/benchmark/retrospective.py +++ b/src/openamp_foundry/benchmark/retrospective.py @@ -3,7 +3,7 @@ Two complementary benchmark modes: STANDARD (primary gate): - Positives — 44 confirmed literature AMPs + Positives — 43 confirmed literature AMPs (unique; REF-GIG-001 duplicate removed in PR #66) Negatives — 44 length-matched peptides drawn from UniProt Swiss-Prot background amino acid frequencies (RNG seed=43) Tests: can the model distinguish AMP-like composition/structure from @@ -12,8 +12,8 @@ Gate: AUROC > 0.70 → proceed; 0.55–0.70 → caution; < 0.55 → STOP. STRICT (order-sensitivity test, secondary): - Positives — 44 confirmed literature AMPs - Negatives — 44 per-sequence composition-matched shuffled decoys (RNG seed=42) + Positives — 43 confirmed literature AMPs + Negatives — 43 per-sequence composition-matched shuffled decoys (RNG seed=42) Tests: does the model have order-dependent signal beyond composition? Expected AUROC: 0.50–0.65 for any composition-heavy scorer. This is intentionally difficult — it tests ONLY the hydrophobic moment term @@ -204,7 +204,7 @@ def run_retrospective_benchmark( labels_ranked = [r["label"] for r in rows] if recall_ks is None: - recall_ks = [10, 20, 44] + recall_ks = [10, 20, 43] recall = {f"recall_at_{k}": round(_recall_at_k(labels_ranked, k), 4) for k in recall_ks} random_auroc = 0.5 diff --git a/src/openamp_foundry/cli.py b/src/openamp_foundry/cli.py index 72612bdf..b351160c 100644 --- a/src/openamp_foundry/cli.py +++ b/src/openamp_foundry/cli.py @@ -542,7 +542,7 @@ def _run_validate_scoring(args: argparse.Namespace) -> int: "auroc_above_random": result["auroc_above_random"], "recall_at_10": result.get("recall_at_10"), "recall_at_20": result.get("recall_at_20"), - "recall_at_44": result.get("recall_at_44"), + "recall_at_43": result.get("recall_at_43"), "interpretation": result["interpretation"], "out": args.out, } @@ -988,7 +988,7 @@ def _score(seq: str) -> dict: "", "This calibration uses the same scoring model as candidate nomination. It is not", "independent validation — it confirms internal consistency, not external predictive power.", - "The AUROC benchmark (AUROC=0.8037 on 44 literature AMPs vs 44 background peptides) is", + "The AUROC benchmark (AUROC=0.8047 on 43 literature AMPs vs 44 background peptides) is", "the appropriate independent validation.", ] diff --git a/tests/test_benchmark_evaluate.py b/tests/test_benchmark_evaluate.py index c6342c05..9610089c 100644 --- a/tests/test_benchmark_evaluate.py +++ b/tests/test_benchmark_evaluate.py @@ -69,9 +69,9 @@ def test_expected_random_recall_k_less_than_positives(self): assert abs(result - 0.1) < 0.001, f"Expected 0.1, got {result}" def test_random_recall_real_benchmark_scale(self): - # 44 AMPs, 88 total candidates, k=8 → E[recall@8] = 8/88 ≈ 0.0909 - result = random_recall_at_k(n_candidates=88, n_positives=44, k=8) - assert abs(result - 8 / 88) < 0.001, f"Expected {8/88:.4f}, got {result}" + # 43 AMPs + 44 background = 87 total; k=8 → E[recall@8] = 8/87 ≈ 0.0920 + result = random_recall_at_k(n_candidates=87, n_positives=43, k=8) + assert abs(result - 8 / 87) < 0.001, f"Expected {8/87:.4f}, got {result}" def test_zero_candidates_returns_zero(self): assert random_recall_at_k(0, 2, 2) == 0.0 diff --git a/tests/test_retrospective.py b/tests/test_retrospective.py index 51f4a3e0..0480e118 100644 --- a/tests/test_retrospective.py +++ b/tests/test_retrospective.py @@ -144,10 +144,11 @@ def test_phase3_benchmark_passes_gate(self): phase3.yaml shifts more weight to safety (0.30 vs 0.25) and synthesis (0.20 vs 0.15), which down-ranks hemolytic AMPs — correct behaviour for a synthesis gate but lowers raw AUROC slightly vs pipeline.yaml. Gate is still AUROC > 0.70 (STRONG). - Measured: AUROC=0.7890 (95% CI 0.6932-0.8784, n=2000 bootstrap; after PR #65 Trp bonus). + Measured: AUROC=0.7846 (95% CI 0.6829-0.8747, n=2000 bootstrap; after PR #66 dedup fix). + Previous values: 0.7936 (before PR #65 Trp bonus), 0.7890 (pre-dedup, post-PR #65). The upper-bound check (< 0.83) is a config-identity sentinel: pipeline.yaml scores - 0.8164 which would breach it, so a silent config fallback is detectable. + 0.8047 which would breach it, so a silent config fallback is detectable. """ from pathlib import Path amp_csv = Path("examples/validation/known_amps.csv") @@ -165,16 +166,17 @@ def test_phase3_benchmark_passes_gate(self): f"phase3.yaml AUROC={result['auroc']:.4f}: synthesis gate does not meet the 0.70 " "threshold. Do not proceed to wet-lab synthesis." ) - # Regression sentinel: measured AUROC=0.7936; alert if it drops below 0.75 + # Regression sentinel: measured AUROC=0.7846 (post-dedup); alert if it drops below 0.75 assert result["auroc"] >= 0.75, ( f"phase3.yaml AUROC={result['auroc']:.4f} < 0.75: a regression has occurred " - "(baseline 0.7936). Synthesis gate still valid at >0.70 but scoring may be degraded." + "(baseline 0.7846). Synthesis gate still valid at >0.70 but scoring may be degraded." ) - # Config-identity sentinel: pipeline.yaml AUROC=0.8164 > 0.83 would fail this, - # so a silent config fallback is caught rather than silently passing with the wrong weights. - assert result["auroc"] < 0.83, ( - f"phase3.yaml AUROC={result['auroc']:.4f} >= 0.83: this is higher than expected " - "for phase3.yaml (0.7936). Check that phase3.yaml (not pipeline.yaml) was loaded." + # Config-identity sentinel: pipeline.yaml AUROC=0.8047 >= 0.795 would fail this, + # while phase3.yaml AUROC=0.7846 < 0.795 passes. Threshold sits between the two measured + # point estimates so a silent config fallback is caught. + assert result["auroc"] < 0.795, ( + f"phase3.yaml AUROC={result['auroc']:.4f} >= 0.795: this is higher than expected " + "for phase3.yaml (measured 0.7846). Check that phase3.yaml (not pipeline.yaml) was loaded." ) print(f"\n[phase3 benchmark] AUROC={result['auroc']:.4f}: {result['interpretation']}") print(f"AUPRC={result.get('auprc', 'N/A')}, Recall@20={result.get('recall_at_20', 'N/A')}")