Iteration 5
After the Self-study of Module 5, you apply what you learned to the project. At the end of this iteration you have better models, and a validation you can trust.
In Iteration 4 you rebuilt your models as pipelines, added the lab database and the test molecules, and looked for the reasons why your test score differs from your cross-validation score. Because your preprocessing is now part of a pipeline, trying another model means changing one step.
In Iteration 5 you work on better models. You have just learned decision trees, random forests and gradient boosting in the Self-study, and you can now try them on the project, together with a validation strategy that resembles the test molecules.
The HOPV molecules were made and tested in real solar cells (Lopez et al., 2016). The lab database and the test molecules come from another source: the Harvard Clean Energy Project (Hachmann et al., 2011), a computational screening of candidate molecules built by combining a few dozen building blocks. The sources differ in ways you may have found:
- The candidate molecules have no side chains of saturated carbon (median
FractionCSP30.00, against 0.11 for HOPV) and are smaller (a median of 28 heavy atoms, against 41). - Many contain selenium or silicon. Selenium is in 83 of the 232 lab molecules and 57 of the 200 test molecules, but in only 1 of the 347 HOPV molecules. The descriptors that are missing for many lab and test molecules are the ones RDKit cannot compute for selenium.
- No test molecule is similar to a HOPV molecule. The median Tanimoto similarity (on Morgan fingerprints) of a test molecule to its most similar HOPV molecule is 0.21; to its most similar lab molecule it is 0.41.
There is a second reason why random cross-validation on HOPV is optimistic: 60% of the HOPV molecules have a close analogue (Tanimoto similarity of at least 0.7) elsewhere in HOPV, so a random fold usually contains a near copy of a validation molecule.
As in Iteration 4, there is no fixed sequence. Keep using your experiment log, and only compare scores that come from the same protocol (see Iterate with a purpose).
Things to investigate
- Validate on molecules like the test molecules. If your Iteration 4 investigation showed that some training molecules resemble the test molecules more than others, build your validation around them, and use the same validation for every model you compare. Cross-validation by group (the Module 3 Self-study) shows how to pass your own folds to
cross_validateandGridSearchCV. - Tree-based models and boosting. Compare a random forest and a gradient boosting model, for example
HistGradientBoostingClassifier, XGBoost or LightGBM, under the same validation. Tune their most important hyperparameters, and use early stopping for boosting. Inside cross-validation, the easiest way is the built-in early stopping ofHistGradientBoostingClassifier(early_stopping=True): it sets aside part of each training fold, so the validation fold stays untouched. Some of these models accept missing values directly: does that help with the lab and test molecules? - Clustering is a tool, not just an exercise. Cluster all molecules together (HOPV, lab database and test set). Which training molecules end up in the same clusters as test molecules? How could you use that: for validation, for choosing or weighting training data, or as a feature?
- Is a difference real? With validation folds of a few dozen molecules, the scores of the folds differ a lot: a standard deviation of 0.04 to 0.09 in macro-F1 is normal here.
RepeatedStratifiedKFoldrepeats the cross-validation with other folds and gives a more stable mean. - Look at your mistakes. Which molecules does your model get most wrong? Do they have something in common: a source, a cluster, a range of a descriptor, missing values?
The Module 5 Self-study uses only XGBRegressor. The classifiers of XGBoost and LightGBM work like the scikit-learn ones, with two differences.
XGBClassifier accepts only class numbers (0, 1, 2), not class names: with low, medium and high it stops with ValueError: Invalid classes inferred from unique values of y. Encode the labels with LabelEncoder, and turn the predictions back into class names before you save a submission, because the scorer expects the names:
from sklearn.preprocessing import LabelEncoder
from xgboost import XGBClassifier
encoder = LabelEncoder()
y_train_encoded = encoder.fit_transform(y_train) # high, low, medium -> 0, 1, 2
model = XGBClassifier().fit(X_train, y_train_encoded)
predictions = encoder.inverse_transform(model.predict(X_test)) # back to class namesLGBMClassifier accepts class names, but prints a warning for every tree it cannot split further, often hundreds of lines in a cross-validation loop. LGBMClassifier(verbose=-1) turns them off.
Go beyond the baseline
Use what you learn throughout the course to explore stronger approaches. Possible directions include:
- new machine-learning algorithms
- feature engineering, for example Morgan fingerprints (
rdkit.Chem.rdFingerprintGenerator) or counts of selected elements such as silicon and selenium - feature selection
- different preprocessing strategies
- alternative validation approaches
- hyperparameter tuning
- ensemble methods, such as voting and stacking
- insights from clustering
- deeper investigation of model errors
The Self-study material provides techniques you can draw from, but you decide which ones are appropriate for the problem.
Score your model on the test molecules when an experiment gives you a better one. In Iteration 6, after the Self-study of Module 6, you work towards your final solution.