Skip to main content

Clustering

Open In Colab

Before you start. Needs: data/002_processed_data/xy_df_hopv.csv, which you create in Data Preparation of Iteration 2. Creates: nothing that later notebooks need.

Paths in this notebook are relative to its folder, modules/3_modeling_and_evaluation/2_datalab/, so the data folder is ../../../data/. If a file is not found, run from pathlib import Path; print(Path.cwd()) to see where the notebook runs, and check that the earlier notebook saved its file in data/002_processed_data/.

In Iteration 1 and in the Baseline notebook of Iteration 2, you clustered molecules with HDBSCAN and looked at the clusters. You did not yet measure how good the clusters are, and you did not choose the number of clusters yourself.

In this notebook you use what you learned in the Clustering notebook of the Self-study: K-Means, the elbow method, the silhouette score and HDBSCAN, on the descriptors of the HOPV molecules. Clustering is not a goal in itself in this project. Look for groups of molecules that help your classification and regression models, for example a cluster where most molecules have a low stability.

# IMPORT WHAT YOU NEED HERE

Step 1 — Prepare the descriptors

  • Load xy_df_hopv.csv into xy_df. Keep the SMILES and the targets aside: you do not cluster on them, but you need them to interpret the clusters.
  • Check the descriptors for missing and infinite values, and decide how to handle them. K-Means cannot handle them at all.
  • Scale the descriptors. Clustering uses distances, and without scaling the descriptors with the largest values decide the clusters, as you saw in Iteration 1.
# YOUR CODE HERE

Step 2 — K-Means: choose K

  • Fit KMeans for K = 2 to 15 (with n_init=10 and a fixed random_state), and plot the inertia against K. Is there a clear elbow?
  • Compute the silhouette score for the same values of K and plot it as well.
  • Choose a value of K and explain your choice. On real data the elbow is often not sharp, and the two methods do not always agree.
# YOUR CODE HERE

Step 3 — HDBSCAN

  • Fit HDBSCAN (with copy=True) on the same scaled descriptors. How many clusters does it find, and how many molecules are noise (label -1)?
  • Compute the silhouette score of the HDBSCAN clusters, without the noise points, and report the share of noise next to it.
  • Compare with K-Means on the same molecules: leaving out the noise points removes the hardest molecules, so compute the K-Means silhouette also without the molecules HDBSCAN calls noise. Which result has the better silhouette score, and which one is easier to interpret?
# YOUR CODE HERE

Step 4 — Interpret the clusters

A good silhouette score does not make a cluster useful. Take the clustering you prefer and look at the molecules in each cluster:

  • How many molecules does each cluster have?
  • Compare the targets per cluster: the share of low, medium and high stability (pd.crosstab), and the mean homo and lumo. Do some clusters differ clearly from the others?
  • Look at the SMILES strings of a few molecules per cluster, as in Iteration 1. What do the molecules of a cluster have in common?

Write down what you found. If a cluster differs in its targets, the cluster label, or the property that defines the cluster, may be a useful feature for your supervised models. Be careful when you use it: a cluster chosen because it stands out in a crosstab of the targets of all molecules has seen the validation molecules, and makes the validation score too optimistic.

# YOUR CODE HERE