# IMPORT WHAT YOU NEED HEREClustering
Before you start. Needs:
data/002_processed_data/xy_df_hopv.csv, which you create in Data Preparation of Iteration 2. Creates: nothing that later notebooks need.Paths in this notebook are relative to its folder,
modules/3_modeling_and_evaluation/2_datalab/, so the data folder is../../../data/. If a file is not found, runfrom pathlib import Path; print(Path.cwd())to see where the notebook runs, and check that the earlier notebook saved its file indata/002_processed_data/.
In Iteration 1 and in the Baseline notebook of Iteration 2, you clustered molecules with HDBSCAN and looked at the clusters. You did not yet measure how good the clusters are, and you did not choose the number of clusters yourself.
In this notebook you use what you learned in the Clustering notebook of the Self-study: K-Means, the elbow method, the silhouette score and HDBSCAN, on the descriptors of the HOPV molecules. Clustering is not a goal in itself in this project. Look for groups of molecules that help your classification and regression models, for example a cluster where most molecules have a low stability.
Step 1 — Prepare the descriptors
- Load
xy_df_hopv.csvintoxy_df. Keep the SMILES and the targets aside: you do not cluster on them, but you need them to interpret the clusters. - Check the descriptors for missing and infinite values, and decide how to handle them. K-Means cannot handle them at all.
- Scale the descriptors. Clustering uses distances, and without scaling the descriptors with the largest values decide the clusters, as you saw in Iteration 1.
# YOUR CODE HEREStep 2 — K-Means: choose K
- Fit
KMeansfor K = 2 to 15 (withn_init=10and a fixedrandom_state), and plot the inertia against K. Is there a clear elbow? - Compute the silhouette score for the same values of K and plot it as well.
- Choose a value of K and explain your choice. On real data the elbow is often not sharp, and the two methods do not always agree.
# YOUR CODE HEREStep 3 — HDBSCAN
- Fit
HDBSCAN(withcopy=True) on the same scaled descriptors. How many clusters does it find, and how many molecules are noise (label-1)? - Compute the silhouette score of the HDBSCAN clusters, without the noise points, and report the share of noise next to it.
- Compare with K-Means on the same molecules: leaving out the noise points removes the hardest molecules, so compute the K-Means silhouette also without the molecules HDBSCAN calls noise. Which result has the better silhouette score, and which one is easier to interpret?
# YOUR CODE HEREStep 4 — Interpret the clusters
A good silhouette score does not make a cluster useful. Take the clustering you prefer and look at the molecules in each cluster:
- How many molecules does each cluster have?
- Compare the targets per cluster: the share of
low,mediumandhighstability (pd.crosstab), and the meanhomoandlumo. Do some clusters differ clearly from the others? - Look at the SMILES strings of a few molecules per cluster, as in Iteration 1. What do the molecules of a cluster have in common?
Write down what you found. If a cluster differs in its targets, the cluster label, or the property that defines the cluster, may be a useful feature for your supervised models. Be careful when you use it: a cluster chosen because it stands out in a crosstab of the targets of all molecules has seen the validation molecules, and makes the validation score too optimistic.
# YOUR CODE HERE