Iteration 6
After the Self-study of Module 6, you apply what you learned to the project. At the end of this iteration you have your final models, scored on the test set, and a write-up of your solution as a portfolio project.
This is the last iteration. You read research on the problem and use its ideas, keep improving your models, score your final models on the test molecules, and write up your solution as a portfolio project.
Apply what you read
In Research to Practice, the last activity of the Self-study, you chose at least one principle from a machine learning paper. Apply it to your project now: decide what you expect to change, run the experiment under your usual validation protocol, and record the result in your experiment log, also when it does not help.
Go further with research
The Self-study and the DataLab gave you the tools, but they don’t cover everything that is known about this problem. Researchers work on the same questions: how to measure energy levels, how to predict them from the structure of a molecule, and how to test such predictions on molecules a model has not seen. The papers below are a starting point, not a reading list to complete. Follow the references that interest you, look for related work (see Understanding ML Research for how to find and read papers), and try ideas that go beyond the course material. Treat each idea as an experiment: decide what you expect to change, test it under your validation protocol and record the result.
Some of the papers are open access. For the others the abstract is free, and a library or the authors’ pages often give access to the full text.
How the targets are measured
Bertrandie, J., Han, J., De Castro, C. S. P., et al. (2022). The Energy Level Conundrum of Organic Semiconductors in Solar Cells. Advanced Materials, 34(35), 2202575. Open access.
The authors determine the energy levels of 33 organic semiconductors with four methods (cyclic voltammetry, density functional theory, and two kinds of photoelectron spectroscopy). The methods give clearly different values, and only some of them agree with how the solar cells perform.
This paper won’t change your models, but it shows how much a target depends on how it is measured. In this course the targets are synthetic, so you take them as given. A real project is more open: you can ask the client how each value was measured, check whether two data sources measured the same thing in the same way, and suggest another measurement method when the current one does not answer the client’s question.
Machine learning for energy levels
Bertrandie, J., Noyan, M. A., Huerta Hernandez, L., et al. (2025). From Experimental Values to Predictive Models: Machine Learning-Driven Energy Level Determination in Organic Semiconductors. Advanced Energy Materials, 16(3), 2403707.
Measured energy levels are expensive to get, so there are few of them. The authors train a chained model: a large database of intermediate properties helps the model learn, and transfer learning carries this over to the small set of measured values. They also test the model on newly measured molecules that were not used in any way during development.
Ideas for your project: could the prediction of one model be a useful feature for another, for example predicted HOMO and LUMO values for stability? If you try it, use out-of-fold predictions (cross_val_predict): a model’s predictions for its own training molecules are too good, and the second model would learn to trust them too much. StackingRegressor in Module 5 does the same for this reason. And how does their external test set compare with the role of the test molecules in your project?
Validation on new kinds of molecules
Wu, Z., Ramsundar, B., Feinberg, E. N., et al. (2018). MoleculeNet: a benchmark for molecular machine learning. Chemical Science, 9(2), 513–530. Open access.
MoleculeNet compares many models on many molecular datasets. Look at how the authors split the data. Besides a random split, they use a scaffold split, which puts molecules with the same core structure in the same part of the split, so the test molecules are structurally different from the training molecules. RDKit computes the scaffold of a molecule (rdkit.Chem.Scaffolds.MurckoScaffold). Look at the scaffolds of your molecules before you use them as groups: a scaffold is the molecule without its side chains, and the lab and test molecules have no side chains, so each of them is its own scaffold. A scaffold split of these molecules is then just a random split. Groups that put similar molecules together work better here: clusters of molecular fingerprints (for example Morgan fingerprints from rdkit.Chem.rdFingerprintGenerator, clustered with the Butina algorithm in rdkit.ML.Cluster.Butina), your clusters from Iteration 5, or the source of the molecules. Pass them as groups to GroupKFold (see Cross-validation by group in the Module 3 Self-study). To check a validation, compare how similar the validation molecules are to the training molecules with how similar the test molecules are to them, for example with the Tanimoto similarity of each molecule to its nearest neighbour in the training data. Does this validation agree better with your test score than your current validation?
Descriptors for organic solar cells
Sahu, H., Rao, W., Troisi, A., & Ma, H. (2018). Toward Predicting Efficiency of Organic Solar Cells via Machine Learning and Improved Descriptors. Advanced Energy Materials, 8(24), 1801032.
The authors predict the efficiency of 280 organic solar cells from 13 physical properties of the molecules, and compare random forests, gradient boosting and neural networks. Their target is different from yours, but the question carries over: does a small set of well-chosen descriptors with a physical meaning work as well as hundreds of generic ones? Compare with the RDKit descriptors you use, and with the features your models find most important.
Iterate and improve
There is no predefined end point. Continue moving through the CRISP-DM cycle as new questions emerge:
understand → prepare → model → evaluate → learn → repeat
You may revisit any part of the workflow, including:
- business and domain understanding
- data understanding
- data preparation
- feature engineering
- feature selection
- validation strategy
- model selection
- hyperparameter tuning
- classification
- regression
- clustering
There is no required order. For example, poor model performance may lead you back to data preparation. Clustering may reveal groups of molecules that behave differently. A validation score that does not agree with your test score may indicate that your validation strategy needs to change.
Your final models
Choose your final models on your validation results, not on the public score: the public score uses only 100 of the 200 test molecules (see Scoring your predictions). When you have chosen, score them with --private. The private score of a model that you chose without looking at it is the honest result to report.