Skip to main content

Classification

Open In Colab

Before you start. Needs: data/002_processed_data/xy_df_hopv.csv, which you create in Data Preparation of Iteration 2. Creates: your models in the models/ folder.

Paths in this notebook are relative to its folder, modules/3_modeling_and_evaluation/2_datalab/, so the data folder is ../../../data/. If a file is not found, run from pathlib import Path; print(Path.cwd()) to see where the notebook runs, and check that the earlier notebook saved its file in data/002_processed_data/.

In the Baseline notebook of Iteration 2, you trained the Iteration 1 classification model on xy_df_hopv.csv with:

  1. a simple train/test split
  2. default model parameters

Your goal in this notebook is to improve that baseline in a controlled way. We will change one thing at a time and track what happens to the performance metric. As in the baseline, report accuracy and macro-F1 side by side, use macro-F1 to make decisions, and compare with the majority-class baseline (DummyClassifier(strategy="most_frequent")).

Load xy_df_hopv.csv into xy_df and remove the rows where stability is missing, as in the baseline.

Save your model(s) at the end.

# IMPORT WHAT YOU NEED HERE

Step 1 — Cross-validation

Relying on a single train/test split can be misleading. By chance, the test set might turn out to be unusually “easy” or “hard,” which could cause you to over- or underestimate your model’s performance. Cross-validation reduces this risk by evaluating the model across multiple splits and then averaging the results, giving a more reliable estimate of performance.

Replace the single train/test split with cross-validation:

  • Use StratifiedKFold(n_splits=5, shuffle=True, random_state=42) from sklearn.model_selection. Stability is a class label, and stratified folds keep the proportions of low, medium and high about the same in every fold.
  • Score accuracy and macro-F1 in one go: cross_validate(model, X, y, cv=cv, scoring=["accuracy", "f1_macro"])
  • Cross-validate the majority-class baseline with the same cv as well
  • Report both the mean and the standard deviation of the test scores

Use the following format with 3 decimals

Metric: Mean ± Standard Deviation (n-fold CV)

For example:

Accuracy: 0.823 ± 0.012 (5-fold CV)

Macro-F1: 0.801 ± 0.015 (5-fold CV)

# YOUR CODE HERE

Compare this result with the previous results. Keep in mind that cross-validation (CV) provides a more reliable estimate of model performance than a single train/test split. A higher score from a single split does not imply that single split is better than CV, it may simply be due to chance. CV is preferred because it gives a more trustworthy estimate of how the model will perform on unseen data, which is ultimately what we care about.

But there are always trade-offs, no gain comes for free. What might be the drawbacks of using cross-validation in practice?

Step 2 — Hyperparameter tuning

We will tune on two key parameters:

  • n_estimators: This controls how many decision trees are in the forest. The default is 100. Adding more trees usually makes the model more stable and accurate, but also increases training time.

  • max_depth: This controls how deep each tree can grow. By default (None), trees keep splitting until every group is pure, which can create very large trees. Shallow trees may underfit (not learn enough), while overly deep trees may overfit (memorize the training data instead of generalizing).

Use RandomizedSearchCV with n_iter=15, with the following hyperparameter values:

param_dist = {
    "n_estimators": [100, 300, 500, 1000],
    "max_depth": [5, 10, 15, 20],
}

Use the cv object you used in step 1, and let the search optimise the project’s metric with scoring="f1_macro". Always fix the random seed for reproducibility.

Our goal here is to practice hyperparameter tuning, not to build the best possible model. You can always improve performance later by tuning more parameters, trying more values, and running longer searches. For now, keep the tuning phase short. Since you will repeat this process many times, faster iterations will help you reach better models in a reasonable amount of time.

Report macro-F1 (the score the search optimised) and accuracy in the same format. For the accuracy, cross-validate the best parameters with the same cv.

Macro-F1: 0.xxx ± 0.xxx (5-fold CV)

Keep in mind that this score is slightly optimistic: the best parameters were chosen on the same CV folds that produced the score.

Also make sure to report and save the best parameters. Once you have found a good set of parameters, you can reuse them to train a model in later iterations without repeating the tuning process every time.

# YOUR CODE

Put everything into a table and reflect. Which change made the biggest difference, and how much do you trust each number? (How many molecules were in the test set of the starter dataset?) Compare the accuracy and the macro-F1 of the baseline: what do they tell you? Your numbers can be slightly different:

Step Molecules Accuracy Macro-F1
starter_dataset.xlsx, simple train/test split, default model parameters 72 0.800 0.801
xy_df, simple train/test split, default model parameters (Iteration 2 baseline) 312 0.635 0.638
xy_df, 5-fold stratified CV, majority-class baseline 312 0.490 ± 0.006 0.219 ± 0.002
xy_df, 5-fold stratified CV, default model parameters 312 0.721 ± 0.041 0.711 ± 0.044
xy_df, 5-fold stratified CV, tuned model parameters 312 0.734 ± 0.030 0.724 ± 0.036

Only the last three rows use the same protocol, so only they can be compared directly. Keep a record of each experiment: the data (and how you filtered it), the features, the model and its parameters, the validation (folds and seed) and the scores. Iteration 4 suggests a simple experiment log.

Save your model(s) into the models folder.

# YOUR CODE HERE
model_name = 'stability_iteration3.joblib'