Skip to content

Feat/unified benchmark cli - #450

Open
Yoko286 wants to merge 24 commits into
mainfrom
feat/unified-benchmark-cli
Open

Yoko286 wants to merge 24 commits into
mainfrom
feat/unified-benchmark-cli

Conversation

@Yoko286

@Yoko286 Yoko286 commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

This pull request synchronizes the local GeochemistryPi CLI workflow with the reproducible scientific workflows validated through the GeochemistryPi MCP wrapper.

The implementation was developed and verified against 12 literature-based geochemical machine-learning benchmarks. Instead of adding paper-specific plotting scripts, this PR extends the general GeochemistryPi framework so that the same analyses can be reproduced through reusable CLI workflows.

Main changes

  • Added reproducible dataset-preparation support for original Excel and CSV datasets.
  • Added deterministic scientific execution controls, including model and train-test split seeds.
  • Improved CLI automation and MCP-to-CLI replay compatibility.
  • Added or improved generic workflows for:
    • regression;
    • classification and confusion matrices;
    • PCA and t-SNE;
    • anomaly detection;
    • continuous geochemical time series;
    • grouped t-SNE followed by hierarchical clustering;
    • embedding and anomaly-label overlays.
  • Improved plot formatting to more closely reproduce the scientific content and presentation of the reference figures.
  • Added preparation configurations and a detailed CLI reproduction ledger.
  • Added validation and regression tests for the newly introduced generic functionality.
  • Excluded generated datasets, run directories, figures, MLflow files, and other reproducible local artifacts from version control.

Literature benchmarks

The local CLI framework was evaluated using the following 12 reproduction targets:

  1. Petrelli et al. (2020), Figures 5a and 6a
  2. Lu et al. (2025), Figures 1a and 1b
  3. Liu et al. (2025), Figure 1b
  4. Ji et al. (2024), Figure 3
  5. White et al. (2025), Figure 4
  6. Tortelli et al. (2026), Figure 2b
  7. Stracke et al. (2022), Figure 5
  8. Tao et al. (2021), Figure 3
  9. Sharapatov et al. (2025), Figure 3a
  10. More et al. (2024), Figure 3a
  11. Keller and Schoene (2012), Figure 1a
  12. Liu et al. (2024), Figure 3a

Validation results

For most benchmarks, the local CLI reproduced the MCP results exactly, including selected numerical tables, metrics, confusion matrices, and principal PNG outputs.

Notable reproduction boundaries:

  • White et al. (2025) was validated using a structural criterion. The six principal mantle clans were recovered, although exact branch lengths and high-order dendrogram ordering remain dependent on the t-SNE seed and software version.
  • More et al. (2024) reproduced the anomaly labels point by point, while the final time-series layout remains an approximation of the published figure.
  • Generated PDFs may differ at the byte level because of document timestamps and other metadata, even when the plotted scientific content is identical.

Detailed datasets, CLI selections, model parameters, hashes, and validation results are documented in:

cli_reproductions/README.md

Reproducibility and repository scope

This PR does not introduce 12 independent paper-specific reproduction scripts. The benchmark papers are used as scientific validation cases for reusable GeochemistryPi capabilities.

Generated inputs, model outputs, figures, MCP run directories, and MLflow tracking data are intentionally excluded from Git because they can be regenerated from the documented source datasets and parameter configurations.

hyunzhou and others added 24 commits August 21, 2026 03:34
…v2' into feat/geochemistrypi-mcp-wrapper-v2

# Conflicts:
#	tests/mcp_wrapper/interaction/test_driver_unit.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants