Regression

Regression in machine learning is the task of predicting a continuous numerical value for each input based on learned patterns from labeled training data. Predicting the price of a house from its features such as size, location, and number of rooms is a regression problem.

Building a regression model is similar to classification in many ways, but there are also key differences. Let’s train our first regression model and see how it compares to what we learned in classification.

1. Your first regression model

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# load the dataset
# the target, y, is continuous, so this is suitable for a regression problem
from sklearn.datasets import load_diabetes
X, y = load_diabetes(as_frame=True, return_X_y=True)

The goal is to predict a quantitative measure of diabetes progression one year after the baseline. It has 10 features such as age, sex, body mass index (BMI), blood pressure, and blood serum measurements.

Remember: unlike classification datasets, the target (y) here is continuous, it can take many values, not just 0 or 1.

# instead of importing RandomForestClassifier
# we import RandomForestRegressor
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(random_state=0)

# same as classification
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# same as classification
model.fit(X_train, y_train)

RandomForestRegressor(random_state=0)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

RandomForestRegressor

?Documentation for RandomForestRegressoriFitted

Parameters

	random_state random_state: int, RandomState instance or None, default=None Controls both the randomness of the bootstrapping of the samples used when building trees (if ``bootstrap=True``) and the sampling of the features to consider when looking for the best split at each node (if ``max_features < n_features``). See :term:`Glossary <random_state>` for details.	0
	n_estimators n_estimators: int, default=100 The number of trees in the forest. .. versionchanged:: 0.22 The default value of ``n_estimators`` changed from 10 to 100 in 0.22.	100
	criterion criterion: {"squared_error", "absolute_error", "poisson"}, default="squared_error" The function to measure the quality of a split. Supported criteria are "squared_error" for the mean squared error, which is equal to variance reduction as feature selection criterion and minimizes the L2 loss using the mean of each terminal node, "absolute_error" for the mean absolute error, which minimizes the L1 loss using the median of each terminal node, and "poisson" which uses reduction in Poisson deviance to find splits, also using the mean of each terminal node. .. versionadded:: 0.18 Mean Absolute Error (MAE) criterion. .. versionadded:: 1.0 Poisson criterion. .. versionchanged:: 1.9 Criterion `"friedman_mse"` was deprecated.	'squared_error'
	max_depth max_depth: int, default=None The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.	None
	min_samples_split min_samples_split: int or float, default=2 The minimum number of samples required to split an internal node: - If int, then consider `min_samples_split` as the minimum number. - If float, then `min_samples_split` is a fraction and `ceil(min_samples_split * n_samples)` are the minimum number of samples for each split. .. versionchanged:: 0.18 Added float values for fractions.	2
	min_samples_leaf min_samples_leaf: int or float, default=1 The minimum number of samples required to be at a leaf node. A split point at any depth will only be considered if it leaves at least ``min_samples_leaf`` training samples in each of the left and right branches. This may have the effect of smoothing the model, especially in regression. - If int, then consider `min_samples_leaf` as the minimum number. - If float, then `min_samples_leaf` is a fraction and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node. .. versionchanged:: 0.18 Added float values for fractions.	1
	min_weight_fraction_leaf min_weight_fraction_leaf: float, default=0.0 The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided.	0.0
	max_features max_features: {"sqrt", "log2", None}, int or float, default=1.0 The number of features to consider when looking for the best split: - If int, then consider `max_features` features at each split. - If float, then `max_features` is a fraction and `max(1, int(max_features * n_features_in_))` features are considered at each split. - If "sqrt", then `max_features=sqrt(n_features)`. - If "log2", then `max_features=log2(n_features)`. - If None or 1.0, then `max_features=n_features`. .. note:: The default of 1.0 is equivalent to bagged trees and more randomness can be achieved by setting smaller values, e.g. 0.3. .. versionchanged:: 1.1 The default of `max_features` changed from `"auto"` to 1.0. Note: the search for a split does not stop until at least one valid partition of the node samples is found, even if it requires to effectively inspect more than ``max_features`` features.	1.0
	max_leaf_nodes max_leaf_nodes: int, default=None Grow trees with ``max_leaf_nodes`` in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.	None
	min_impurity_decrease min_impurity_decrease: float, default=0.0 A node will be split if this split induces a decrease of the impurity greater than or equal to this value. The weighted impurity decrease equation is the following:: N_t / N * (impurity - N_t_R / N_t * right_impurity - N_t_L / N_t * left_impurity) where ``N`` is the total number of samples, ``N_t`` is the number of samples at the current node, ``N_t_L`` is the number of samples in the left child, and ``N_t_R`` is the number of samples in the right child. ``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum, if ``sample_weight`` is passed. .. versionadded:: 0.19	0.0
	bootstrap bootstrap: bool, default=True Whether bootstrap samples are used when building trees. If False, the whole dataset is used to build each tree.	True
	oob_score oob_score: bool or callable, default=False Whether to use out-of-bag samples to estimate the generalization score. By default, :func:`~sklearn.metrics.r2_score` is used. Provide a callable with signature `metric(y_true, y_pred)` to use a custom metric. Only available if `bootstrap=True`. For an illustration of out-of-bag (OOB) error estimation, see the example :ref:`sphx_glr_auto_examples_ensemble_plot_ensemble_oob.py`.	False
	n_jobs n_jobs: int, default=None The number of jobs to run in parallel. :meth:`fit`, :meth:`predict`, :meth:`decision_path` and :meth:`apply` are all parallelized over the trees. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary <n_jobs>` for more details.	None
	verbose verbose: int, default=0 Controls the verbosity when fitting and predicting.	0
	warm_start warm_start: bool, default=False When set to ``True``, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new forest. See :term:`Glossary <warm_start>` and :ref:`tree_ensemble_warm_start` for details.	False
	ccp_alpha ccp_alpha: non-negative float, default=0.0 Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ``ccp_alpha`` will be chosen. By default, no pruning is performed. See :ref:`minimal_cost_complexity_pruning` for details. See :ref:`sphx_glr_auto_examples_tree_plot_cost_complexity_pruning.py` for an example of such pruning. .. versionadded:: 0.22	0.0
	max_samples max_samples: int or float, default=None If bootstrap is True, the number of samples to draw from X to train each base estimator. - If None (default), then draw `X.shape[0]` samples irrespective of `sample_weight`. - If int, then draw `max_samples` samples. - If float, then draw `max_samples * X.shape[0]` unweighted samples or `max_samples * sample_weight.sum()` weighted samples. .. versionadded:: 0.22 .. versionchanged:: 1.9 Float `max_samples` is relative to `sample_weight.sum()` instead of `X.shape[0]` for weighted samples.	None
	monotonic_cst monotonic_cst: array-like of int of shape (n_features,), default=None Indicates the monotonicity constraint to enforce on each feature. - 1: monotonically increasing - 0: no constraint - -1: monotonically decreasing If monotonic_cst is None, no constraints are applied. Monotonicity constraints are not supported for: - multioutput regressions (i.e. when `n_outputs_ > 1`). Read more in the :ref:`User Guide <monotonic_cst_gbdt>`. .. versionadded:: 1.4	None

Fitted attributes

Name	Type	Value
estimator_ estimator_: :class:`~sklearn.tree.DecisionTreeRegressor` The child estimator template used to create the collection of fitted sub-estimators. .. versionadded:: 1.2 `base_estimator_` was renamed to `estimator_`.	DecisionTreeRegressor	DecisionTreeRegressor()
estimators_ estimators_: list of DecisionTreeRegressor The collection of fitted sub-estimators.	list	[DecisionTreeR...ate=209652396), DecisionTreeR...ate=398764591), DecisionTreeR...ate=924231285), DecisionTreeR...te=1478610112), ...]
estimators_samples_ estimators_samples_: list of arrays The subset of drawn samples (i.e., the in-bag samples) for each base estimator. Each subset is defined by an array of the indices selected. .. versionadded:: 1.4	list	[array([236, 2..., dtype=int32), array([107, 1..., dtype=int32), array([301, ..., dtype=int32), array([163, 2..., dtype=int32), ...]
feature_importances_ feature_importances_: ndarray of shape (n_features,) The impurity-based feature importances. The higher, the more important the feature. The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance. Warning: impurity-based feature importances can be misleading for high cardinality features (many unique values). See :func:`sklearn.inspection.permutation_importance` as an alternative.	ndarray[float64](10,)	[0.05,0.01,0.31,...,0.02,0.31,0.06]
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0	ndarray[object](10,)	['age','sex','bmi',...,'s4','s5','s6']
n_features_in_ n_features_in_: int Number of features seen during :term:`fit`. .. versionadded:: 0.24	int	10
n_outputs_ n_outputs_: int The number of outputs when ``fit`` is performed.	int	1

# Same as classification,
# but now the output values are continuous numbers.
y_pred = model.predict(X_test)
y_pred

array([249.77, 237.06, 173.44, 110.9 , 214.83, 268.97,  96.11, 210.42,
       145.59, 245.27, 173.51, 146.31, 131.37,  99.04, 274.61,  84.17,
       166.55,  82.06, 116.32, 208.43, 174.46, 134.4 , 179.6 , 120.47,
       207.11, 193.46, 146.17,  74.95, 194.32, 168.62, 182.36,  73.27,
       117.48, 150.84, 140.29, 171.14, 167.36, 149.67,  94.26, 213.24,
       107.22, 164.27, 117.81, 186.5 , 192.07,  87.66, 129.8 , 134.4 ,
       109.53, 236.78, 160.93,  67.58, 155.66, 162.49, 240.94, 197.51,
       178.46,  85.3 , 126.57, 163.15, 240.76, 145.89, 123.13,  96.55,
       266.96, 156.54,  97.13, 227.93, 215.67, 118.82,  83.68, 147.32,
       124.48, 119.23, 126.9 , 181.13, 123.59, 225.33, 245.61, 215.59,
       119.85, 186.75,  69.72, 239.35, 111.09,  92.45, 151.87, 194.59,
        99.81])

This changes how we evaluate the model. A simple way to see how well it performs is to plot predictions vs. true labels on a scatter plot.

plt.scatter(y_test, y_pred)
plt.xlim(0, 350)
plt.ylim(0, 350)
plt.plot([0, 350], [0, 350], label='perfect prediction line')
plt.xlabel("Labels (y_test)")
plt.ylabel("Predictions (y_pred)")
plt.legend()

Points close to the line are great (perfect if exactly on the line). Points far from the line are poor. To measure performance, we can check the average difference between predictions and true values, which is the error.

error = y_pred - y_test
error

362   -71.23
249    22.06
271    46.44
435    46.90
400    39.83
       ...  
381     7.09
213    43.45
134    48.87
49     52.59
52     40.81
Name: target, Length: 89, dtype: float64

We could take the average error, but positive and negative values would cancel out. For example, +100 and -100 average to 0, which hides the real error.

# very low comparing to the errors we see in the plot
# which is due to positive and negative errors cancelling each other
np.mean(error)

np.float64(3.534269662921351)

To fix this, we take the absolute value of each error.

np.abs(error)

362    71.23
249    22.06
271    46.44
435    46.90
400    39.83
       ...  
381     7.09
213    43.45
134    48.87
49     52.59
52     40.81
Name: target, Length: 89, dtype: float64

Then we take the mean, this gives us an error value that actually makes sense.

np.mean(np.abs(error))

np.float64(47.75224719101124)

this is called the mean absolute error (MAE). You can tell from the code:

np.mean(np.abs(error))

and you guessed it right, it is already implemented in scikit-learn

YOUR TURN

Calculate MAE using scikit-learn’s implemented method, you should obtain the same value you obtained above.

# YOUR CODE HERE

My MAE was 47.75, yours might differ slightly due to randomness.

2. Baselines

Is this a good model? What does a MAE of ~48 even mean?

In classification, model performance is easier to interpret, for example, an accuracy of 90% sounds good because we know that for a binary and balanced problem, random guessing gives about 50% accuracy (like flipping a coin). So, 90% is clearly much better than chance.

In regression, interpretation depends on the domain. If you’re predicting house prices, a $1,000 error might be excellent, while a $100,000 error would be terrible. The meaning of the error depends entirely on the scale of the target variable.

When we don’t know the domain, a useful baseline is to predict the average of the training targets for every test data point and measure the resulting MAE. This baseline acts like the “coin flip” in classification, if your model doesn’t perform better than this, it isn’t learning anything meaningful.

# the average of y_train
# like a model, the baseline may only learn from the training data
y_train.mean()

np.float64(151.60623229461757)

# predict everything as the average of y_train
y_pred_baseline = np.ones_like(y_test)*y_train.mean()
y_pred_baseline

array([151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229, 151.60623229, 151.60623229, 151.60623229,
       151.60623229])

from sklearn.metrics import mean_absolute_error
mae_baseline = mean_absolute_error(y_test, y_pred_baseline)
mae_baseline

59.22745647261036

My baseline error was 59.23, and the model’s error was 47.75, much better! So, the model is clearly learning some useful relationships. A perfect model would achieve a MAE of zero, so this model is still far from a perfect model.

# how predicting every datapoint as the mean looks like
plt.scatter(y_test, y_pred_baseline)
plt.xlim(0, 350)
plt.ylim(0, 350)
plt.plot([0, 350], [0, 350], label='perfect prediction line')
plt.xlabel("Labels (y_test)")
plt.ylabel("Predictions (y_pred)")
plt.legend()

3. Exercise

YOUR TURN

Train and test a regression model based on the California Housing Dataset to predict the price of a house from its properties

from sklearn.datasets import fetch_california_housing
X, y = fetch_california_housing(as_frame=True, return_X_y=True)

# YOUR CODE HERE

What is the MAE of your model in 💵? It should be roughly $30–40K. Always check the documentation to understand what the target variable (y) actually represents.