Skip to content

Fix custom splits during preprocessing - #208

Merged
jan-forest merged 4 commits into
devfrom
bugfix/custom_splits
Sep 14, 2026
Merged

jan-forest merged 4 commits into
devfrom
bugfix/custom_splits

Conversation

@ChristianBaeuchl

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes the handling of user-provided custom_splits during preprocessing.

Changes

_utils.py

  • Added custom_splits_from_anno() function.
    • Reads an annotation file and converts a column containing split labels into the dict format expected by the splitter.
    • Sorts the annotation table by the sample_id column to align the resulting row indices with the sample IDs that are sorted during preprocessing.

_base_pipeline.py

  • Updated BasePipeline.
    • Passes custom split information to the preprocessor used by the base pipeline.

_base_preprocessor.py

  • Updated BasePreprocessor.
    • Added a custom_splits attribute to the constructor.
    • Passes the custom_splits information to PairedUnpairedSplitter.

_datasplitter.py

  • Updated PairedUnpairedSplitter.
    • Added a custom_splits attribute to the constructor.
    • Updated split() so that it checks whether custom splits exist.
    • Added _per_modality_splits_from_custom_splits().
      • Checks custom split existence and required keys.
      • Takes the annotation index as the source of sample IDs.
      • Converts each split from row indices to sample-ID sets.
      • Checks for overlap between splits.
      • Creates empty split containers for every modality.
      • Maps custom split IDs to each data modality.
      • Checks whether any custom-split samples are missing from all modalities.
      • Returns sample-ID sets per modality.

Testing

Tested locally with:

  • an Imagix IMG_TO_IMG example using 2D MedMNIST PNG data and an annotation file with a custom_splits column
  • an Imagix3D IMG_TO_IMG example using predefined annotation-based train/validation/test splits

In both cases, preprocessing used the custom split path via PairedUnpairedSplitter.

@jan-forest
jan-forest merged commit 16c3e2a into dev Sep 14, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants