Advanced Scikit-learn

In this notebook we explore some of the more advanced features of scikit-learn. You can work with scikit-learn for a long time without ever using these ideas, and most people do not include them in their day to day code. Still, once you learn them, they will make you more efficient and capable as a data scientist.

Outline

1. Pipelines
2. Column Transformers
3. Tuning a Pipeline
4. Exercise
5. Additional Concepts

Working in Colab? This notebook uses optuna, which Colab does not include. Run the cell below: it installs the package only when it detects Colab. Locally it does nothing, because the course environment already includes optuna.

# Installs the missing packages on Google Colab only; does nothing locally.
import sys

if "google.colab" in sys.modules:
    %pip install -q optuna

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

1. Pipelines

📘 Hands-On Machine Learning with Scikit-Learn, Keras, & TensorFlow, Géron, 3rd Ed. >You got introduced to the Pipeline in Chapter 2 of the course textbook (section Transformation Pipelines p.83). In this section, you’ll learn it again from scratch and deepen your understanding. Please review this section quickly as a refresher. —

What is a Pipeline and why should we use one? Let’s understand this by looking at an example. We will repeat the very first case from the notebook 005_data_prep.ipynb, where we scaled the data before training a support vector machine.

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.svm import SVR
from sklearn.metrics import mean_absolute_error
from sklearn.preprocessing import StandardScaler

X, y = fetch_california_housing(as_frame=True, return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# how X_train looks before scaling
X_train.head()

# scale
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)

# how X_train looks after scaling
pd.DataFrame(X_train, columns=scaler.feature_names_in_)

# train
model = SVR()
model.fit(X_train, y_train)

# we cannot use X_test directly
# we must apply the same scaler that was fitted on X_train
X_test.head()

# if we skip scaling here
# the model will receive inputs in the wrong scale and perform poorly
y_pred = model.predict(np.array(X_test))
mae = mean_absolute_error(y_test, y_pred)
print(mae)

# apply the scaler that was fitted on X_train
# you shouldn't fit a new scaler here, the scaling was already learned
X_test = scaler.transform(X_test)

# test
y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(mae)

The issue here is the preprocessing and the model are separate pieces that you must manage yourself. Saving the model alone is not enough. When new data arrives, you must apply exactly the same fitted transformers, so you also need to save the scaler and any other preprocessing steps.

As your workflow grows, this becomes difficult to track, and you will eventually end up writing your own functions to bundle these steps together. There is also the problem that when everything is developed separately, each part is tuned in isolation while the rest is kept constant. In many cases, jointly optimising the entire chain leads to better results.

Now let’s repeat the example using a pipeline.

# repeat
X, y = fetch_california_housing(as_frame=True, return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

from sklearn.pipeline import Pipeline

pipeline = Pipeline(steps=[
                ('scaler', StandardScaler()),
                ('model', SVR())])

pipeline

Now these two steps are combined into a single pipeline. We fit the pipeline as one unit, so we no longer fit the scaler and the SVR separately. When making predictions, we also avoid scaling first and predicting second. We can pass the raw data directly into the pipeline and it will handle everything in the correct order. With only two steps the benefit may seem small, but as your preprocessing becomes more complex, the convenience of letting the pipeline manage the entire chain becomes obvious.

# We can use X_train without scaling
X_train.head()

# this runs all preprocessing steps and then fits the model in one call
pipeline.fit(X_train, y_train)

# we can now use X_test in its raw form
# the pipeline will handle the scaling internally instead of us doing it manually
X_test.head()

# the pipeline applies preprocessing and prediction in one step
y_pred = pipeline.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(mae)

Now that we have seen what a pipeline does and why it is useful, let’s look at its formal definition. The documentation states:

A sequence of data transformers with an optional final predictor. Pipeline allows you to sequentially apply a list of transformers to preprocess the data and, if desired, conclude the sequence with a final predictor for predictive modeling. Intermediate steps of the pipeline must be transformers, that is, they must implement fit and transform methods. The final estimator only needs to implement fit.

This means, a pipeline can be imagined as an ordered chain:

transformer1 → transformer2 → ... → transformerN → model (optional)

You can include as many transformers as you want, or even build a pipeline with only transformers and no model at the end.

The pipeline we built looks like this:

StandardScaler → SVR

which can be seen in the flowchart Jupyter Notebook creates.

YOUR TURN

Click on the flowchart and find the value of the hyperparameter C used in SVR.

C is the regularization strength in an SVM and it controls how closely the model tries to fit the training data. Regularization is a way to prevent overfitting by limiting how flexible the model can be. A large value of C allows the model to fit the training data very tightly, trying hard to avoid mistakes, while a small value of C encourages a smoother and simpler decision boundary by allowing more training errors.

# interact with the flowchart
pipeline

Let’s get familiar with the pipeline instance.

# check the type of the pipeline object
type(pipeline)

# view all steps in the pipeline by their assigned names
pipeline.named_steps

# access a specific step in the pipeline by its name
pipeline['model']

# number of input features the pipeline saw when it was fitted
pipeline.n_features_in_

# names of the input features the pipeline saw during fitting
pipeline.feature_names_in_

# get all parameters of the pipeline
pipeline.get_params()

Why do the parameter names look like model__C instead of just C?

Once we use a pipeline, we have several steps and each step has its own parameters. If we used just C, the pipeline would not know which step that parameter belongs to.

The documentation explains that the pipeline lets us set parameters for each step by using the step name, two underscores __, and then the parameter name.

The purpose of the pipeline is to assemble several steps that can be cross-validated together while setting different parameters. For this, it enables setting parameters of the various steps using their names and the parameter name separated by a ’__’.

Remember when we created the pipeline above, we used the step names scaler and model:

pipeline = Pipeline(steps=[
                ('scaler', StandardScaler()),
                ('model', SVR())])

This means, if we want to set the C hyperparameter of SVR, we must refer to it as model__C. In other words, we always use <step_name>__<parameter_name> when working with pipeline parameters.

We can set parameters directly when building the pipeline by passing them into the steps, or we can update the parameters later using the pipeline’s parameter naming scheme.

# set C=10 directly when constructing the pipeline
pipeline = Pipeline(steps=[
    ('scaler', StandardScaler()),
    ('model', SVR(C=10))
])

YOUR TURN

Click on the flowchart and find the value of the hyperparameter C used in SVR. Verify that it is 10.

pipeline

or after construction by combining the step name and param name:

# or set C=10 after the pipeline is created
pipeline = Pipeline(steps=[
                ('scaler', StandardScaler()),
                ('model', SVR())])

pipeline = pipeline.set_params(model__C=10)

YOUR TURN

Click on the flowchart and find the value of the hyperparameter C used in SVR. Verify that it is 10.

pipeline

you can update the parameters of the pipeline during training by chaining

pipeline = Pipeline(steps=[
                ('scaler', StandardScaler()),
                ('model', SVR())])

# update and train
pipeline.set_params(model__C=10).fit(X_train, y_train)

y_pred = pipeline.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(mae)

Note that the pipeline gives the same MAE as our manual approach with the same parameters. We are not changing how the model behaves. We are only using this approach to bundle several steps into one pipeline. (The last MAE is different only because we set C=10.)

YOUR TURN

Create a pipeline with MinMaxScaler and SVR. Use the default parameters while defining the pipeline. After creating the pipeline, update the following parameters:

• MinMaxScaler: set feature_range to (-1, 1), the default is (0, 1)
• SVR: set C to 0.1, the default is 1.

Train and test this model on the housing dataset.

X, y = fetch_california_housing(as_frame=True, return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# YOUR CODE HERE

You can save and load the pipelines using joblib, similar to saving and loading models.

So far so good, however notice that a transformer always applies its operation to the entire input. When our feature matrix contains different types of columns that need different preprocessing, this approach is not enough. To handle this properly, we will now introduce the ColumnTransformer.

2. Column Transformers

Now let’s set up a problem with heterogeneous data, solve it once without pipelines, and then solve it again using pipelines with column transformers. To create this scenario, I will modify the housing dataset by turning the HouseAge column into a categorical feature HouseAgeCat and introducing some random missing values in the MedInc column. This will require us to use an ordinal encoder for HouseAgeCat and an imputer for MedInc.

X, y = fetch_california_housing(as_frame=True, return_X_y=True)

# turn HouseAge into a categorical feature named HouseAgeCat
# then drop the original numeric column
X['HouseAgeCat'] = pd.qcut(X['HouseAge'], q=3, labels=['new', 'mid', 'old'])
X = X.drop(columns=['HouseAge'])
X.head(n=10)

# introduce missing values into the MedInc column (about 10 percent of the rows)
rng = np.random.default_rng(0) # fixed seed for reproducible results
mask = rng.random(len(X)) < 0.10
X.loc[mask, 'MedInc'] = np.nan
X.head(n=10)

Now let us see how we might handle this without using pipelines. You have already seen encoding and imputation in 005_data_prep.ipynb.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# impute missing values in the MedInc column using the median
# fit the imputer on the training data and apply the same transformation to the test data
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
X_train[["MedInc"]] = imputer.fit_transform(X_train[["MedInc"]])
X_test[["MedInc"]] = imputer.transform(X_test[["MedInc"]])

# encode the categorical HouseAgeCat column using the defined order
# fit the encoder on the training data and apply the same transformation to the test data
from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder(categories=[['new', 'mid', 'old']], dtype='int')
X_train["HouseAgeCat"] = encoder.fit_transform(X_train[["HouseAgeCat"]])
X_test["HouseAgeCat"] = encoder.transform(X_test[["HouseAgeCat"]])

# MedInc now has no missing values
# HouseAgeCat is encoded as integers
X_train.head(n=10)

Now let’s train a RandomForestRegressor:

from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(random_state=0)
model.fit(X_train, y_train)

and test:

y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(mae)

We obtained a reasonable MAE, so the approach works. Our next goal is to reproduce the same process using pipelines, which will require the ColumnTransformer.

From the documentation:

Applies transformers to columns of an array or pandas DataFrame. This estimator allows different columns or column subsets of the input to be transformed separately and the features generated by each transformer will be concatenated to form a single feature space. This is useful for heterogeneous or columnar data, to combine several feature extraction mechanisms or transformations into a single transformer.

This means, it creates a transformer that applies specific transformations to selected columns while passing the remaining columns through unchanged. This allows us to preprocess only the columns we choose without affecting the rest of the data.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# create a custom transformer that applies different preprocessing to different columns
# it takes X as input, transforms only the specified columns, and leaves the rest unchanged
# this can be plugged directly into a pipeline
from sklearn.compose import ColumnTransformer
ct = ColumnTransformer(
    transformers=[
        ("ordinal_enc", OrdinalEncoder(categories=[['new', 'mid', 'old']], dtype='int'), ["HouseAgeCat"]),
        ("imputer", SimpleImputer(), ["MedInc"])]
)

# now we can use it like any other transformer
# fit it on the training data and apply the transformations
ct.fit_transform(X_train)

But there is a problem. We passed the entire X_train to the transformer, yet the output only contains the transformed columns. We want the full X_train back, with only the selected columns modified. To achieve this, we can use the remainder parameter, which tells the transformer what to do with the columns that are not explicitly listed.

ct = ColumnTransformer(
    transformers=[
        ("ordinal_enc", OrdinalEncoder(categories=[['new', 'mid', 'old']], dtype='int'), ["HouseAgeCat"]),
        ("imputer", SimpleImputer(), ["MedInc"])
    ],
    remainder='passthrough' # add this
)

ct.fit_transform(X_train)

Now the full X_train is returned, with only the selected columns transformed and the rest left untouched. We can now place this column transformer inside a pipeline.

pipeline = Pipeline(steps=[
    ('preprocess', ct),
    ('model', RandomForestRegressor())
])

YOUR TURN

Inspect the pipeline flowchart, given below. Find the answers to the following, on the flowchart:

• What is the name of the column that goes into the imputer?
• What is the value of the strategy parameter in the imputer?

pipeline

# you can access all the params
# including the ones inside the ct
pipeline.get_params()

In our example without the pipelines, we used random_state=0 and strategy='median'. We need to set these for our pipeline to follow the same steps. The MAE can still differ slightly (in the fourth decimal), because the ColumnTransformer puts the transformed columns first and a random forest with a fixed random_state depends on the column order.

YOUR TURN

Update the parameters of the pipeline, by setting random_state=0 for the random forest and strategy='median' for the imputer. Inspect the flowchart to ensure the parameters were set correctly.

Solution will be given below.

# YOUR CODE HERE

# my solution
pipeline = pipeline.set_params(
    preprocess__imputer__strategy='median',
    model__random_state=0
)

pipeline

pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(mae)

pipeline.feature_names_in_

3. Tuning a Pipeline

As a final step, let us see how a pipeline can be tuned. This is where joint optimization becomes useful. For example, suppose we want to find out whether mean or median imputation works better. So far our workflow has been the following: run cross validation with mean imputation while keeping the model hyperparameters fixed, then run cross validation with median imputation while keeping the model hyperparameters fixed, and pick the better one. After that, we would tune the model hyperparameters while keeping the chosen strategy fixed.

With a pipeline, we no longer need to separate these steps. We can tune the entire pipeline at once and let cross validation jointly select both the imputation strategy and the model hyperparameters.

import optuna
from sklearn.model_selection import cross_validate

# recreate the pipeline as before
# this time we set random_state=0 directly when defining the model
pipeline = Pipeline(steps=[
    ('preprocess', ct),
    ('model', RandomForestRegressor(random_state=0))
])

# now we tune the entire pipeline
# the process is the same as tuning a model, but parameter names include the step name
# for example: model__n_estimators or preprocess__imputer__strategy
def objective(trial):
    params = {
        "model__n_estimators": trial.suggest_int("model__n_estimators", 50, 200),
        "model__min_samples_split": trial.suggest_int("model__min_samples_split", 2, 10),
        "preprocess__imputer__strategy": trial.suggest_categorical("preprocess__imputer__strategy", ["mean", "median"])
    }

    # update params
    p = pipeline.set_params(**params)

    # cross validate
    result = cross_validate(p, X_train, y_train, cv=3, scoring='neg_mean_absolute_error')
    mae = np.mean(result['test_score'])*-1

    return mae

study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=10)

print("Best params:", study.best_params)
print("Best mae:", study.best_value)

# apply the best parameters found by Optuna
# you can verify in the flowchart that n_estimators matches the selected value
pipeline.set_params(**study.best_params)

pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(mae)

4. Exercise

YOUR TURN

Create a pipeline with MinMaxScaler and SVR. Tune the pipeline on the housing dataset.

X, y = fetch_california_housing(as_frame=True, return_X_y=True)

# YOUR CODE HERE

5. Additional Concepts

Scikit-learn is a large library and there is not enough time to cover every idea in detail. As a data scientist, it is important to become comfortable learning new concepts when they are needed, because it is impossible to know everything in advance. Below is a short list of topics worth being aware of. You do not need to master them now, but it helps to know they exist so that you can return to them later when they become relevant.

• Pipelines and composite estimators
• Combining predictors using stacking
• Semi-supervised learning
• Anomaly detection
• Data generation (make_classification, make_regression etc.)
• Plotting calibration curves
• Nested cross-validation
• and take a look at the User Guide for more.