Gaia benchmarks

This is the public, benchmark-only evaluation package for Gaia. The three fully executed notebooks are the main documentation:

Notebook	What it explains and reproduces	Evaluation script
01_cohort_level_benchmarks.ipynb	Drug perturbation priors, all 44 observed trial ORRs and their clinical sources, predicted-versus-observed ORR, comparisons, and controls	scripts/cohort_level.py
02_patient_level_benchmarks.ipynb	CRC-first patient evidence with exact sample/slide/GSM/FOV lineage, measured on-treatment separation, pretreatment prediction, measured alignment, and cSCC transfer with its linkage limit	scripts/patient_level.py
03_biology_level_benchmarks.ipynb	The old frozen 289-row BiologyBench protocol, biological references, scoring rules, KO gene-direction audit, and results	scripts/biology_level.py

The related public project page is alunalabs/Gaia. Model availability is tracked there; this repository does not contain model weights.

scripts/model_benchmark_runner.py is the model-facing execution path used in all three notebooks. The model itself is a small adapter; score construction and benchmark logic remain public.

Results at a glance

Benchmark	Released result	Important qualification
Cohort predicted ORR	Pearson 0.650, Spearman 0.594, MAE 10.25 ORR percentage points on 44 disease–drug rows	Both axes are on the ORR scale; every observed value is traceable to the source ledger
Historical clinical prior	Pearson 0.465, Spearman 0.460, MAE 11.26 points on the same 44 rows	Uses historical clinical ORR summaries rather than modeled tumor biology
Cell-line drug sensitivity	Pearson −0.014 on 40 covered rows	Relative sensitivity rank, not an ORR estimate
CRC measured on-treatment gene panel	ROC AUC 0.867 on 11 patients; related signed-risk view 0.900	Biological positive control on actual treated tissue, not a pretreatment prediction; same-cohort analysis
CRC pretreatment patient response	ROC AUC 0.800 on 11 patients	Exact control: 29/462 class-balanced label assignments reached or exceeded the real AUC (p=0.0628)
CRC predicted-versus-measured biology	Best descriptive mean cosine 0.304 at rollout step 4	Uses measured on-treatment tissue and selects the best step on the same 11 patients
cSCC pretreatment patient response	ROC AUC 0.778 on 12 patients	Secondary transfer benchmark; exact control 61/924 (p=0.0660); raw score scale is not calibrated to CRC
BiologyBench KO	257/289 = 88.9% direction passes	All row-level KO expectations are receiver-readout decreases
BiologyBench OE	257/281 = 91.5% direction passes	Eight of 289 OE rows are not applicable
BiologyBench KO gene-direction audit	0/173 audited ligand rows had every observed pathway gene decrease	Median 61.9% decreased; 10 ligand rows and all 106 regional rows lack this exact audit

These small benchmarks describe the included evaluation surfaces; they are not estimates of clinical utility.

Trace a result back to its source

The notebooks expose the retained lineage rather than only citing a study at the end. Detailed dataset and sample linkage is included for the patient-level benchmark:

patient source study or public data record
  → included specimen, section, patient code, and timepoint
  → compact benchmark input or observed target
  → public score construction
  → reported metric

scripts/provenance.py checks that the 44 cohort ORR targets are covered by 63 clinical source records, that all 11 CRC patients have exact paired sample lineage, and that the cSCC subset retains its stated study-level linkage boundary. Detailed patient/sample provenance is confined to the patient-level benchmark.

Cohort results are on the ORR scale

The cohort notebook compares gaia_predicted_orr_pct directly with observed objective response rate (ORR) for 44 disease–drug pairs. Both quantities are percentage points, so correlation, MAE, RMSE, and the calibration diagonal have clear units. The notebook does not present a unitless gene-readout score versus observed ORR.

Every target expands to its source trial or study record, source URL, reported ORR, and evaluable denominator where available. When several fully denominated records support one target, the notebook reconstructs the denominator-weighted ORR.

The benchmark primarily uses broad, disease-level all-comer trial results. The spatial patient samples used as upstream model contexts often lack the complete biomarker labels required to construct a faithful biomarker-enriched proxy cohort. Rather than assign unknown labels or imply a matched enriched population, the benchmark evaluates against broadly enrolled disease cohorts. Some historical source arms still have their own eligibility criteria, so “primarily all-comer” does not mean every source was literally unselected or that these scores estimate efficacy in a named biomarker-positive subgroup.

What a drug perturbation prior means

A prior is a gene-indexed additive expression-shift vector derived from treated single cells versus vehicle controls matched within cell line and experimental plate. Gene deltas are averaged within cell line and then equally across cell lines. The vector is applied to selected cells in measured tissue before a matched spatial rollout. It is not an embedding, fold change, clinical label, or training target. The cohort notebook includes the formula, a schematic, and the boundary of what the compact public export can reproduce.

CRC-first patient evidence

The patient notebook begins with a biological positive control on measured on-treatment CRC tissue. Six declared gene modules cover KRAS–MAPK output, EGFR engagement, proliferation, EMT/invasion, hypoxia/stress, and apoptosis. They form four zero-centered failure risks; each is converted to no-failure support with sigmoid(−risk/0.10), and their geometric mean is the response- oriented panel score. This fully reproducible panel reaches ROC AUC 0.867 on 11 patients. A related retained signed-risk view reaches 0.900.

This is evidence that the benchmark endpoint has recognizable biological content, not evidence of prospective prediction: measured on-treatment tissue already contains treatment response, the cohort is small, and the analysis is not independent validation. The complete gene lists, score code, patient rows, and caveats sit together in the notebook.

Shared pretreatment score contract

CRC and cSCC use the same final patient-level arithmetic over four bounded gene-expression supports:

bio_conversion_support = mean(response_conversion_support,
                              terminal_efficacy_support)

universal_gene_readout_score = coverage_support
                               * bio_conversion_support
                               * resistant_tail_control

Therapy-specific adapters supply the inputs: CRC uses apoptosis for terminal efficacy, while checkpoint-treated cSCC uses control of persistent exhaustion and suppressive inflammatory escape. Labels are used only for ROC AUC.

The raw scores are not cross-cohort probabilities. cSCC conversion is nearly saturated while terminal support is near zero for most patients, so the arithmetic mean stays near 0.5 and the final values cluster near 0.3. CRC has several near-zero coverage or resistant-tail gates, producing scores near zero. The repository checks both formulas and the cSCC source export exactly.

BiologyBench and the KO monotonicity check

BiologyBench contains 183 ligand–receptor pathway statements and 106 regional tissue-program statements. A row passes when the edited sender program moves the spatial receiver readout in its prespecified direction. Ligand/receptor vocabulary is organized from CellChat/CellChatDB, and ligand-row pathway activity uses signed PROGENy weights.

The separate KO gene-direction table checks a narrow monotonic artifact. Across 173 ligand rows with archived per-gene detail, no row had every observed PROGENy pathway gene decrease; the median fraction decreasing was 61.9%. For the worked colon EGFR row, the pathway score decreased by 0.966 while 155 of 301 pathway genes decreased and 146 did not. This rules out the explanation that those audited passes arise because every receiver pathway gene was forced down. It does not rule out every possible global-shift confounder, and it is not generalized to rows without the same audit.

Data files

Path	Contents
data/cohort_orr.csv	One ORR-scale result per disease–drug benchmark target
data/cohort_trial_evidence.csv	One source result per trial/study record, covering all 44 observed ORRs
data/patient_crc_response.csv	CRC labels, four score gates, and current patient score
data/patient_crc_observed_ontreatment_panel.csv	CRC measured-tissue risk axes, reconstructed supports, panel score, and companion signed-risk score
data/patient_crc_panel_modules.csv	Complete named CRC gene-module definitions and roles
data/patient_cscc_response.csv	cSCC labels, four score gates, and current patient score
data/patient_crc_module_vectors.csv	Predicted and measured CRC program changes by patient, module, and rollout step
data/patient_sample_provenance.csv	Exact CRC sample/slide/GSM/FOV mappings and the available study-level cSCC linkage
data/biology_benchmark.csv	All 289 BiologyBench statements, literal deltas, signed scores, and pass fields
data/biology_benchmark_ko_gene_directions.csv	Per-row KO pathway-gene direction counts for 173 audited ligand rows

scripts/benchmark_metrics.py contains readable, dependency-free metric implementations. Run all numerical reports and provenance checks with:

python scripts/run_all_benchmarks.py

Run the notebooks

The notebooks are checked in with outputs and render on GitHub. To execute them:

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
jupyter nbconvert --to notebook --execute --inplace 01_cohort_level_benchmarks.ipynb
jupyter nbconvert --to notebook --execute --inplace 02_patient_level_benchmarks.ipynb
jupyter nbconvert --to notebook --execute --inplace 03_biology_level_benchmarks.ipynb

Run against a model adapter

Implement the BenchmarkModel protocol in scripts/model_benchmark_runner.py, expose a zero-argument factory, and select it as your_module:create_model. The runner accepts CSV or JSONL cases, preserves adapter-specific extra columns, and writes newly computed benchmark rows. For example:

PYTHONPATH=. python scripts/model_benchmark_runner.py biology   --adapter your_module:create_model   --cases your_biologybench_cases.jsonl   --output new_biologybench_results.csv

The included released-replay adapter is only a deterministic smoke test of the orchestration and scoring path; it reads checked-in output columns and is not fresh inference. The notebooks say this explicitly next to the code cells that exercise it.

Reproduction boundary

Every Python file in scripts/ is an evaluation or metric-reproduction script. There is no training, checkpoint-building, cloud-launch, or model-fitting code. The included files reproduce formulas, metrics, controls, tables, and figures from compact exported results; they do not rerun upstream tissue inference.

License: non-commercial use only

Copyright © 2026 Aluna.

Repository-authored material is licensed under the Creative Commons Attribution-NonCommercial 4.0 International license (CC BY-NC 4.0). Commercial use is not permitted under this license. Underlying studies, databases, and source datasets retain their own terms; see NOTICE.md.