Skip to content

fix: resolve CRITICAL pre-wetlab pipeline issues (SEED-008 excluded, benchmark config mismatch) - #61

Merged
cschanhniem merged 2 commits into
mainfrom
feat/pre-wetlab-polish-3
Jun 28, 2026
Merged

fix: resolve CRITICAL pre-wetlab pipeline issues (SEED-008 excluded, benchmark config mismatch)#61
cschanhniem merged 2 commits into
mainfrom
feat/pre-wetlab-polish-3

Conversation

@cschanhniem

@cschanhniem cschanhniem commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two CRITICAL issues found by pre-synthesis code review:

  1. SEED-008 (puroindoline-a) completely excluded from synthesis pool — The Boman index uses W=-3.398 (protein-binding potential), which correctly captures electrostatic cationic AMPs but systematically penalises Trp-rich interfacial-insertion AMPs like puroindoline-a. The phase3.yaml max_disagreement: 0.30 threshold excluded SEED-008 parent and all variants (disagreement 0.37-0.42). Raised to 0.40 (matches pipeline.yaml and the benchmarked configuration). Result: SEED-008 now contributes 24 selected candidates.

  2. SEED-007/008/009/010 never generated — The previous phase3 run only used SEED-001 through SEED-006. All four seeds added after that run (including the two new seeds from PR feat: AUPRC benchmark metric, 2 new seeds (Bac2A + KR-12), assay pre-registration #58/59) had zero candidates in the synthesis pool. Regenerated phase3_pool.csv and phase3 outputs with all 10 seeds.

Changes

  • configs/phase3.yaml: max_disagreement: 0.30 → 0.40, with scientific rationale in notes (Wimley-White interfacial W=-1.85 vs Boman protein-binding W=-3.398)
  • examples/sequences/phase3_pool.csv: regenerated with all 10 seeds → SEED-007=27, SEED-008=24, SEED-009=18 selected; SEED-010 correctly fails safety gate (KR-12 cytotoxic risk)
  • src/openamp_foundry/benchmark/retrospective.py: validate canonical AA before calling synthesis_feasibility_score
  • src/openamp_foundry/benchmark/evaluate.py: require EF>1 in top-50% (not just any k) for benchmark verdict
  • src/openamp_foundry/scoring/activity.py: fix cross_bonus normalizer comment to describe pH 7.4 charge density
  • tests/test_pipeline_filters.py: update 2 tests that encoded the old (wrong) assumption that phase3 disagrement is stricter than pipeline

Test plan

  • 1183 tests passing
  • SEED-008 parent (FPVTWRWWKWWKG) disagreement=0.3751 — now passes 0.40 gate ✓
  • KDKDKDKD disagreement=0.73 — correctly blocked by both configs ✓
  • phase3 is still stricter on safety (max_safety_risk 0.40 vs 0.70) ✓
  • phase3_pool.csv contains all 10 seed families ✓

🤖 Generated with Claude Code

@cschanhniem
fix: resolve CRITICAL pre-wetlab pipeline issues found by code review
7e54f96 
[CRITICAL] phase3.yaml: raise max_disagreement 0.30 → 0.40 to allow
SEED-008 (puroindoline-a Trp-rich scaffold) through synthesis gate.
The Boman index W=-3.398 measures protein-binding potential, not AMP
interfacial insertion (Wimley-White W=-1.85). Disagreement 0.37-0.42
on Trp-rich sequences reflects mechanism divergence, not uncertainty.
The 0.30 threshold systematically excluded an entire AMP mechanism class.

[CRITICAL] examples/sequences/phase3_pool.csv: regenerate with all 10 seeds.
The previous phase3 run used only SEED-001 through SEED-006. SEED-007
(bombolitin-II), SEED-008 (puroindoline-a), SEED-009 (Bac2A), SEED-010
(KR-12) were never generated. After raising disagreement threshold:
SEED-007=27 selected, SEED-008=24 selected, SEED-009=18 selected.
SEED-010 (KR-12) correctly fails safety gate (KR-12 has cytotoxic risk).

[MEDIUM] retrospective.py: validate canonical AA before calling
synthesis_feasibility_score instead of hardcoding valid_sequence=True.

[MEDIUM] evaluate.py: fix benchmark verdict to require EF>1 in top-50%
of results (not just any k), preventing trivially passing rankers.

[LOW] activity.py: update cross_bonus normalizer comment to describe
pH 7.4 charge density (0.25-0.30) not the old proxy estimate (0.30).

[TEST] test_pipeline_filters.py: update 2 tests that encoded the old
(incorrect) assumption that phase3 is stricter than pipeline on
disagreement. phase3 is now stricter on SAFETY only (max_safety_risk
0.40 vs 0.70), not disagreement (both now 0.40).
@cschanhniem
fix: address code review findings from PR #61
317bcab 
- evaluate.py: guard verdict against silent false-negative when all
  explicit ks exceed n//2 — fall back to full results list in that case [HIGH]
- retrospective.py: move canonical AA check before compute_features call
  so non-canonical sequences don't produce indeterminate feature dicts [MEDIUM]
- phase3.yaml: fix Boman W description from 'protein-binding potential'
  to 'most hydrophobic; lowest protein-interaction potential' [MEDIUM]
- test_pipeline_filters.py: fix stale PPPPPPPP docstring (still said
  'above phase3 gate (0.30)' after we raised it to 0.40) [MEDIUM]
- test_pipeline_filters.py: add SEED-008 disagreement pin test that
  asserts the mechanism-divergence zone value (0.30-0.40) so future
  changes to Boman W or aromatic bonus will be caught immediately [MEDIUM]
@cschanhniem
cschanhniem merged commit f1dde04 into main Jun 28, 2026
2 checks passed
@cschanhniem cschanhniem mentioned this pull request Jun 28, 2026
Merged
2 tasks done
cschanhniem added a commit that referenced this pull request Jun 28, 2026
@cschanhniem
docs: update discovery prediction for PR #61 synthesis pool changes (#62
d5052f4 
)

1184 tests passing. Honest update to calibrated probability: 10-18% (6 confirmed scaffolds) instead of 18-30% (estimated 10). Documents correct synthesis pool after PR #61 regeneration.
@cschanhniem cschanhniem mentioned this pull request Jun 28, 2026
Merged
4 tasks done
cschanhniem added a commit that referenced this pull request Jun 28, 2026
@cschanhniem
fix: correct stale documentation found in post-PR#61 audit (#64)
9c7059a 
- WET_LAB_HANDOFF.md: update max_disagreement for phase3.yaml from 0.30→0.40;
  both configs now use 0.40 since PR #61 reinstated SEED-008 (Trp-rich puroindoline-a)
- METHODS.md §4.4 + §9: correct "45-sequence" → "72-sequence" reference set
  (amp_curated_references.csv has 72 entries; 45 was pre-expansion count)
- DISCOVERY_PREDICTION.md: correct "45-sequence" → "72-sequence" in novelty section;
  fix SEED-005 exemplar from variant KRFFKKIGSALKFA → actual seed KRLFKKIGSALKFL
  with note that VAR_009 (KRFFKKIGSALKFA) is the best-ranked variant
@cschanhniem cschanhniem mentioned this pull request Jun 28, 2026
Merged
3 tasks done
cschanhniem added a commit that referenced this pull request Jun 28, 2026
@cschanhniem
feat: pre-wet-lab polish — error handling, CLI validation, WAVE2 cont…
bb5c18e 
…ext (#67)

* feat: pre-wet-lab polish — error handling, validation, CHANGELOG, WAVE2 context

CRITICAL fix:
- cli.py pilot-panel: bare JSONDecodeError on malformed JSONL → structured error with
  line number and 120-char line preview; returns exit code 1 (was unhandled crash)

MEDIUM fixes:
- cli.py validate-scoring stdout now includes n_positives, n_negatives, benchmark_type,
  auprc alongside auroc (previously needed --out file to see these fields)
- cli.py synthesis-order: explicit CSV header validation before row iteration; missing
  candidate_id/sequence columns → structured error with found_columns list
- WAVE2_PLAN.md: added SEED-008 Trp mechanism divergence note (disagreement ~0.37 is a
  mechanistic marker — Boman W=-3.398 vs Wimley-White W=-1.85 kcal/mol — not uncertainty;
  safe to expand Trp-rich scaffold in Wave 2 if SEED-008 hits)

HIGH fix:
- CHANGELOG.md Unreleased section updated with PRs #61-66 impact; stale "phase3.yaml: 0.30"
  (wrong) corrected to 0.40 (unified in PR #61)

LOW fix:
- presynth_check.py check_sequence(): validates all 20 canonical amino acids before QC;
  raises ValueError with bad-residue list on non-canonical input

Tests added (1191 → 1200):
- test_presynth_qc.py::TestCheckSequenceInputValidation (6 tests)
- test_cli.py::test_validate_scoring_stdout_includes_n_and_auprc
- test_cli.py::test_pilot_panel_malformed_jsonl_returns_error
- test_cli.py::test_synthesis_order_missing_columns_returns_error

* fix: remove overpromised min-length claim from CHANGELOG; annotate dataset-size assertions
This was referenced Jun 28, 2026
Merged
Merged
cschanhniem added a commit that referenced this pull request Jun 28, 2026
@cschanhniem
docs: update DISCOVERY_PREDICTION.md with PR #72 metrics (#73)
a727426 
- Pipeline AUROC 0.8348→0.8420 (bootstrap CI₉₅: 0.76–0.91)
- Phase3 AUROC 0.8126→0.8266; AUPRC 0.8443→0.8627
- max_disagreement gate 0.40→0.45 in all narrative text
- SEED-008 Trp-rich disagreement updated 0.37→~0.43
- Summary table header updated to "After PRs #61–72 (current)"
- Recall@20 updated 43%→44%
@cschanhniem cschanhniem mentioned this pull request Jun 28, 2026
Merged
3 tasks done
cschanhniem added a commit that referenced this pull request Jun 28, 2026
@cschanhniem
chore: update ROADMAP.md metrics and add coverage gate (#74)
46a4f79 
* chore: update ROADMAP.md with current metrics, add coverage gate to make coverage

- AUROC 0.8164→0.8420 (bootstrap CI₉₅: 0.76–0.91) in v0.4 milestone entry
- Add v0.6.x section documenting PRs #61–#72 scoring improvements
- Mark AUPRC gap as Done (PR #58, pipeline AUPRC=0.8627)
- Update large-scale benchmark gap row with corrected n=87 and CI
- coverage target: add --cov-fail-under=80 so make coverage exits non-zero
  below 80%; currently passes at 87.40%

* docs: fix v0.6.x chain gap (add PR #65) and AUPRC attribution
This was referenced Jun 28, 2026
Merged
Merged
cschanhniem added a commit that referenced this pull request Jun 28, 2026
@cschanhniem
docs: sync pre-lab documents with 7-scaffold pool state (#77)
fad1084 
* docs: sync pre-lab docs with 7-scaffold pool (PRs #61-76)

- docs/NOMINATION_REPORT.md: add historical-record banner directing readers
  to current outputs/ for the post-PR #72 synthesis pool
- reports/external_predict.py: document Macrel v1.6.0 ONNX model bug where
  log-softmax scores are compared against 0.5 (causing ALL sequences to be
  classified as NAMP, including canonical AMPs like magainin-2 and LL-37);
  recommend web server as reliable alternative and add --keep-negatives flag
  to CLI instructions

* fix: correct Macrel v1.6.0 bug description and URL per review

- Use package-documented URL (big-data-biology.org/software/macrel) instead
  of unverified subdomain
- Change "range [-1, 0]" to "always <= 0; observed ~-0.8 to -0.05 on
  canonical AMPs" (log-softmax is bounded above by 0 but not below)
- Update module docstring to match web-server-preferred stance and link to
  bug note
cschanhniem added a commit that referenced this pull request Jun 28, 2026
@cschanhniem
fix: update DISCOVERY_PREDICTION.md scaffold count 6→7 post-PR #72 (#82)
fd59f58 
The External Calibration section said "6 effective scaffolds" after PR #61,
but PR #72 (face_segregation_bonus) reinstated SEED-001 (magainin-1) into the
synthesis pool as the 7th scaffold family. Update the calibrated probability
calculation accordingly:

  P(≥1 from 7 at 1.6%) ≈ 11%, at 2.4% ≈ 16%, at 3.1% ≈ 20%
  Calibrated range: 10–20% (was 10–18% for 6 scaffolds)

Also:
- Update executive summary to show post-PR #72 numbers (AUROC 0.8420, 7 families)
- Add cross-reference to docs/WET_LAB_PROBABILITY.md for per-family breakdown
- Clarify "publishable novel result" vs "breaking news" distinction in the table
- Fix "calibrated estimate n=6" table column header → n=7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cschanhniem