A green solvent screening tool for emerging materials via uncertainty aware, transformer enhanced transfer learning

This repository includes the code base of the paper "A green solvent screening tool for emerging materials via uncertainty aware, transformer enhanced transfer learning".

Table of Contents

1. Summary
2. Prerequisites
3. Parameter and Input Specification
4. How to create the data
5. How to Run
6. Expected Inputs and Outouts
7. Troubleshooting and Bugfixing

Summary

Our pipeline consists of

• A customly altered pre-trained foundational model (DimeNet) that outputs a vector for each molecule
• A transformer model
• A Gaussian processes model
• Optional Chemical Description

All options are configurable in the parameters.json file

Prerequisites

Tested Configuration:

• Operating System: Ubuntu 22.04 (jammy)
• IDE: Visual Studio Code

Note: As a rule, everything should work on other systems as well but we cant guarantee it.

• The specific library versions we used with can be found in the requirements.txt. They can be installed directly using pip install -r requirements.txt from the terminal on the folder level that the requirements.txt is. The code will also benefit from CUDA should you have an NVIDIA graphics card

Parameter configuration

The pipeline can be deeply altered by diving into the source code. Nevertheless, a user friendly approach to change the model architecture and training is through altering the parameters.json file.

parameters.json parameters

Parameter	Type	Description
prediction_target	List	a list of strings containing the names of the properties you wish to create predictions for (as named in the data folder)
dimenet_target	List	a list of properties used to pretrained DimeNet to be used. Can take any subset (excluding the empty set) of the values [0, 1,2, 5, 6, 7, 8, 9, 10, 11]
mean_module	List	a list of gaussian processes mean modules to be used. Currently supported : "LinearMean", ConstantMean", "ZeroMean"
standardization	List	a list of boolean values that determines whether to standardize the values of the targets or not.
uncertainty	List	a list of boolean values that determines whether to use Gaussian Processes as the final layer of the model or not. It is a list because it can be sweeped as any other hyperparameter.
log_flag	Bool	true or false depending on if the values of the target should be processed by a log10 preprocessing function. Useful when the target spans many orders of magnitude
divided_by_10_flag	Bool	true or false depending on if the values of the target should be divided by 10 before processing. Experimental.
transfomers_flag	Bool	rue or false depending on whether a transformer layer should be used between the foundational model and the Gaussian processes
nr_of_mixtures	Integer	Currently the Gaussian Processes Kernel function is fixed to Gaussian Mixtures. This value determines how many mixtures this kernel should have. Can only be an integer
descriptor_flag	Bool	true or false depending on whether you want to use molecular descriptors along with the foundational model outputs
save_folder	String	a string denoting the folder where the model results will be saved. If the folder doesnt exist it will be automatically generated
validation	Bool	true or false depending on whether you want to use a validation set for early stopping. Useful against overfitting in large datasets, detrimental in very small ones.
normalize_descriptors	Bool	true or false denoting if the descriptors will be normalized or not
transformer_heads	Integer	an integer denoting the number of the transformer heads. -1 will use as many heads as the feature dimensionality of the transformer layer input
solvent_prediction_flag	Bool	true or false depending on whether you want to perform predictions on the large VOC dataset
dimenet_flag	Bool	true or false depending on whether you want to use the pretrained foundational model or not
keep_all_descriptors	Bool	true or false depending on whether you want to keep all descriptors calculated.
output_dim	Integer	a value denoting the dimensionality of the features entering the trainable part of the pipeline Only relevant if keep_all_descriptors is false. It discards every descriptor beyond the number given, -7 if dimenet is used. i.e. if dimenet is used, there are 30 descriptors and this value is 17 the last 20 descriptors will be discarded
keep_all_descriptors	Bool	true or false depending on whether you want to keep all descriptors calculated
EPOCHS	Integer	number of training epochs
nr_of_runs	Integer	number of folds in the k-fold validation

How to Create the data

• Navigate into the data_creation_pipeline folder. Install the requirements.txt of that folder running pip install -r requirements.txt from the terminal on the folder level that the requirements.txt is.
• The data should be given in the format of a csv, much like the exemplary DN.csv found in that folder.(a column with a name of the chemical and a column named after the property with the property values (names of columns should have no spaces))
• in the graph_parameters.json you can give the list of properties and the corresponding csv that they will be found
• for ease we have included the VOC dataset in the graph_in folder (and for double ease we didnt use git lfs).
• adjust in the graph_parameters.json to your needs (explanatory table below)
• run the file main_full_graph_creation.py (python main_full_graph_creation.py from the terminal on that folder level)

graph_parameters.json parameters

Parameter	Type	Description
property	List	a list of strings containing the names of the properties you wish to create graphs for
file	List	a list of file that contain the corresponding properties
dimenet_target	List	a ;list of integers containing the dimenet targets you want to embed in the graphs
VOC_prediction_flag	Bool	true or false depending on whether you wish to create the relevant descriptors and embeddings for the VOC dataset
initial_processing	Bool	true or false depending on wether you wish to create the initial graphs of the properties mentioned from the relevant csv (to pre-empt doing this multiple times)
descriptor_processing	Bool	true or false depending on if you want to create the descriptors for the properties mentioned (to pre-empt doing this multiple times)
dimenet_outputs	Bool	true or false depending on if you want to embed the dimenet outputs for the properties mentioned (to pre-empt doing this multiple times)
copy_data_to_model_input_dir	Bool	true or false depending on whether you wish to copy the generated graphs to the folder the model expects the inputs to be (overwrites data of the same name existing in that folder)

If you want to see which descriptors are chosen, in the graph_out folder you will find a folder named after each property you processed (property_graphs, e.g. AN_graphs). There you will find a lot of .pt files and one descriptor_names.txt. In the txt you will find the chosen descriptors. Their meanings can then be found from ochem

If you wish to further expand the solvent_graphs do the following steps:

• get a csv with a list of molecule names
• create a column with a dummy property and fill it with a random non zero number
• run the main_full_graph_creation pointing to that file and property, setting in graph_parameters.json:
  • initial_processing : true
  • descriptor_processing: false
  • dimenet_outputs : false
• copy and paste the results in the solvent_graphs in the graph_in folder.

Note that since we get the information from pubchem, molecules with incomplete information from pubchem or molecues that dont appear at all wont be processed so you might end up with fewer graphs than molecules.

For simplicity of testing, the data can be also downloaded from

https://syncandshare.lrz.de/getlink/fiUHBk5ULNmtNTfL3ZBqMV/

and need to be placed in the data folder.

How to Run

• Open the terminal and navigate to the folder level that contains the main.py file.
• Activate the environment with the libraries if you have created one. If you have installed the requirements on the default Path, ignore this step.
• Type python main.py and press enter

Expected Inputs and Outputs

As input the model expects

• the parameters.json adjusted to your specification
• the graphs in the data folder (they should be in the form of graphs and stored in a .pt file).
  • if you only want to train your model: only the data of your case study
  • if you want to additionally create predictions for the VOC dataset, those are expected too

As output the model will give:

• a true vs prediction graph of the k-fold test set
• a file containing the concatenated predictions, truths and standard deviations of all tests of the kfold process
• if you have selected VOC dataset prediction , one file for the means and standard deviations of the VOC dataset for each fold run.

Troubleshooting and Bugfixing

If you notice any bugs, have trouble with our code or have a suggestion for a feautre/extention please contact us at Johnkouroudis[at]gmail.com or directly through git.