Classification

In Iteration 1, you trained a baseline classification model using:

starter_dataset.xlsx with 80 molecules
a simple train/test split
default model parameters

Your goal in this notebook is to improve that baseline in a controlled way. We will change one thing at a time and track what happens to the performance metric.

Save your model(s) at the end.

# IMPORT WHAT YOU NEED HERE

Step 1 — More data

Load xy_df_hopv.csv into xy_df and use this processed dataset of 347 molecules. Train and test the model, exactly as you did in Iteration 1, so that the only change is the dataset.

# YOUR CODE HERE

Compare the test accuracy to the one you obtained in Iteration 1 (Classification). Did it improve, stay the same, or decrease? Provide a possible explanation for the outcome.

Step 2 — Cross-validation

Relying on a single train/test split can be misleading. By chance, the test set might turn out to be unusually “easy” or “hard,” which could cause you to over- or underestimate your model’s performance. Cross-validation reduces this risk by evaluating the model across multiple splits and then averaging the results, giving a more reliable estimate of performance.

Replace the single train/test split with cross-validation:

Use KFold(n_splits=5, shuffle=True, random_state=42)
Report both the mean and the standard deviation of the test scores

Use the following format with 3 decimals

Metric: Mean ± Standard Deviation (n-fold CV)

For example:

Accuracy: 0.823 ± 0.012 (5-fold CV)

# YOUR CODE HERE

Compare this result with the previous results. Keep in mind that cross-validation (CV) provides a more reliable estimate of model performance than a single train/test split. A higher score from a single split does not imply that single split is better than CV, it may simply be due to chance. CV is preferred because it gives a more trustworthy estimate of how the model will perform on unseen data, which is ultimately what we care about.

But there are always trade-offs, no gain comes for free. What might be the drawbacks of using cross-validation in practice?

Step 3 — Hyperparameter tuning

We will tune on two key parameters:

n_estimators: This controls how many decision trees are in the forest. The default is 100. Adding more trees usually makes the model more stable and accurate, but also increases training time.
max_depth: This controls how deep each tree can grow. By default (None), trees keep splitting until every group is pure, which can create very large trees. Shallow trees may underfit (not learn enough), while overly deep trees may overfit (memorize the training data instead of generalizing).

Use RandomizedSearchCV with n_iter=15, with the following hyperparameter values:

param_dist = {
    "n_estimators": [100, 300, 500, 1000],
    "max_depth": [5, 10, 15, 20],
}

Use the cv object you used in step 2. Always fix the random seed for reproducibility.

Our goal here is to practice hyperparameter tuning, not to build the best possible model. You can always improve performance later by tuning more parameters, trying more values, and running longer searches. For now, keep the tuning phase short. Since you will repeat this process many times, faster iterations will help you reach better models in a reasonable amount of time.

Report the accuracy using the same format:

Accuracy: 0.911 ± 0.011 (5-fold CV)

Keep in mind that this score is slightly optimistic: the best parameters were chosen on the same CV folds that produced the score.

Also make sure to report and save the best parameters. Once you have found a good set of parameters, you can reuse them to train a model in later iterations without repeating the tuning process every time.

# YOUR CODE

Put everything into a table and reflect. Which change made the biggest difference, and how much do you trust each number? (How many molecules were in the test set of the starter dataset?) Your numbers can be slightly different:

Step	Accuracy
starter_dataset.xlsx, simple train test split, default model parameters	0.500
xy_df, simple train test split, default model parameters	0.603
xy_df, 5-fold CV, default model parameters	0.688 ± 0.036 (5-fold CV)
xy_df, 5-fold CV, tuned model parameters	0.704 ± 0.051 (5-fold CV)

Save your model(s) into the models folder.

# YOUR CODE HERE
model_name = 'stability_iteration2.joblib'