import matplotlib.pyplot as plt
import numpy as np
import pandas as pdRandom Forests
Once you understand decision trees, it becomes much easier to understand random forests. A random forest is a meta-estimator that trains many decision trees. It is called a random forest, because it uses a random subset of samples and features. By combining their predictions, averaging for regression or majority vote for classification, the random forest improves predictive accuracy and reduces overfitting compared to a single decision tree.
Note that each tree in the forest is trained on a random subset of the samples (bootstrapping). By default, however, each tree still has access to all features when fitting to its subset of the data. The additional randomness comes from the fact that at each split, the tree does not consider all features but instead evaluates only a random subset of them.
The modern form of Random Forests was introduced by Leo Breiman in 2001 in his landmark paper Random Forests. It was published in the journal Machine Learning. It combined previous ideas such as the CART algorithm we described previously and Bagging Predictors, again a paper by Breiman published in 1996.
Random Forests were the dominant algorithms for tabular prediction tasks in the early years of Kaggle, before 2014-2015. They provided strong performance with minimal tuning and were widely used by top competitors.
1. A Random Forest from Decision Trees
Let’s fit 9 decision trees, each trained on a different subset of the dataset. Together, these trees form a small random forest. Note that we will use all the features, since we only have two.
# DUMMY DATA GENERATION
# DO NOT MODIFY
np.random.seed(2) # fixed seed so that everyone gets the same data
n = 200
X = np.random.rand(n, 2) * 2
y = np.zeros(n, dtype='int')
y[(X[:, 0] > 1) & (X[:, 1] > 1)] = 1from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)plt.figure(dpi=100)
plt.scatter(X_train[y_train==0][:, 0], X_train[y_train==0][:, 1], label='class_0')
plt.scatter(X_train[y_train==1][:, 0], X_train[y_train==1][:, 1], label='class_1')
plt.xlabel('feature_1')
plt.ylabel('feature_2')
plt.legend()we can create a simple function to sample the dataset:
def sample_subset(X, y, ratio):
'''
Sampling with replacement
This is how sampling is described in Bagging Predictors, L. Breiman, 1996
'''
n_samples = int(len(X)*ratio)
idx = np.random.choice(len(X), size=n_samples, replace=True)
return X[idx], y[idx]# run this cell multiple times
X_sub, y_sub = sample_subset(X_train, y_train, 0.2)
plt.figure(dpi=100)
plt.scatter(X_sub[y_sub==0][:, 0], X_sub[y_sub==0][:, 1], label='class_0')
plt.scatter(X_sub[y_sub==1][:, 0], X_sub[y_sub==1][:, 1], label='class_1')
plt.xlabel('feature_1')
plt.ylabel('feature_2')
plt.legend()Now we can construct our random forest: we loop n_estimators times, draw a random bootstrap sample of the dataset, and train a decision tree on each sample.
from sklearn.tree import DecisionTreeClassifier
n_estimators = 9
random_forest = []
for i in range(n_estimators):
X_sub, y_sub = sample_subset(X_train, y_train, 0.3)
dt = DecisionTreeClassifier(random_state=0, max_depth=2, max_features=2) # max_features=2 we use all the features
dt.fit(X_sub, y_sub)
random_forest.append(dt)random_forestwe can visualize the forest, and verify each tree has different feature <= th values
from sklearn.tree import plot_tree
n_estimators = len(random_forest)
rows, cols = 3, 3
fig, axes = plt.subplots(rows, cols, figsize=(15, 15), dpi=100)
axes = axes.flatten() # make indexing easier
for i, (tree, ax) in enumerate(zip(random_forest, axes)):
ax.set_title(f"Tree {i+1}")
plot_tree(
tree,
feature_names=['feature_1', 'feature_2'],
class_names=['class_0', 'class_1'],
filled=True,
rounded=True,
fontsize=10,
ax=ax
)
# If you have only 5 trees, hide the last empty subplot
for j in range(i+1, len(axes)):
axes[j].axis("off")
plt.tight_layout()
fig.suptitle(f'A Random Forest with {n_estimators} trees')
plt.show()To understand how a random forest makes a final prediction, let’s look at the predictions of each individual tree. We take one test sample, pass it through all trees, and collect their outputs.
# Each tree makes a prediction on the first test data point
feature_1, feature_2 = X_test[0]
preds = []
for tree_id, dt in enumerate(random_forest):
pred = dt.predict([[feature_1, feature_2]])
preds.append(pred[0])
print(f"Tree {tree_id} predicts: {pred[0]}")# we can calculate the majority vote
majority_vote = int(sum(preds) / len(preds) > 0.5)
majority_voteWe looked at the first prediction, there might be full agreement between trees or not. Next, let’s explicitly search for a test sample on which the trees do not all agree, so we can see how the random forest combines conflicting predictions:
for i, (feature_1, feature_2) in enumerate(X_test):
preds = []
for tree_id, dt in enumerate(random_forest):
pred = dt.predict([[feature_1, feature_2]])
preds.append(pred[0])
if sum(preds) / len(preds) == 0:
continue # all trees agree it is a 0
elif sum(preds) / len(preds) == 1:
continue # all trees agree it is a 1
else:
# means they do not agree
disagreement_id = i
break# Let's check the disagreement
feature_1, feature_2 = X_test[disagreement_id]
preds = []
for tree_id, dt in enumerate(random_forest):
pred = dt.predict([[feature_1, feature_2]])
preds.append(pred[0])
print(f"Tree {tree_id} predicts: {pred[0]}")majority_vote = int(sum(preds) / len(preds) > 0.5)
majority_vote# check if the prediction is correct
y_test[disagreement_id] == majority_voteand finally let’s repeat this process for all the test points and calculate the test accuracy:
y_pred = []
for i, (feature_1, feature_2) in enumerate(X_test): # loop each test datapoint
preds = []
for tree_id, dt in enumerate(random_forest): # loop each tree on a given datapoint
pred = dt.predict([[feature_1, feature_2]])
preds.append(pred[0])
majority_vote = int(sum(preds) / len(preds) > 0.5)
y_pred.append(majority_vote)from sklearn.metrics import accuracy_score
acc = accuracy_score(y_test, y_pred)
print("Accuracy:", acc)2. RandomForestClassifier
Let’s replicate what we have done above, with RandomForestClassifier
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(random_state=0, n_estimators=9, max_features=2, max_depth=2)
rf.fit(X_train, y_train)this is how you can reach the trees
rf.estimators_n_estimators = len(rf.estimators_)
rows, cols = 3, 3
fig, axes = plt.subplots(rows, cols, figsize=(15, 15), dpi=100)
axes = axes.flatten() # make indexing easier
for i, (tree, ax) in enumerate(zip(rf.estimators_, axes)):
ax.set_title(f"Tree {i+1}")
plot_tree(
tree,
feature_names=['feature_1', 'feature_2'],
class_names=['class_0', 'class_1'],
filled=True,
rounded=True,
fontsize=10,
ax=ax
)
# If you have only 5 trees, hide the last empty subplot
for j in range(i+1, len(axes)):
axes[j].axis("off")
plt.tight_layout()
fig.suptitle(f'A Random Forest with {n_estimators} trees')
plt.show()y_pred = rf.predict(X_test)
acc = accuracy_score(y_test, y_pred)
print("Accuracy:", acc)You may notice that the accuracy differs between our earlier implementation and this one. Why could that happen?
One of the most common attributes of a random forest is feature_importances_. This tells us how important each feature is for making predictions across the entire forest.
A single decision tree computes feature importance by looking at how much each split reduces impurity. You have already seen that every split chooses a feature and threshold that maximally reduces Gini impurity. For that tree, the importance of a feature is the total impurity reduction contributed by all splits that use that feature. Deep splits count less than early splits, because they affect fewer samples.
A random forest simply averages these importance values across all trees. Features that consistently lead to large impurity reductions (especially near the top of trees) receive high importance scores. Features that are rarely chosen, or whose splits do not reduce impurity much, get small scores.
This makes feature_importances_ a useful, model-specific measure of how strongly each feature contributes to the forest’s decisions.
Given the symmetry in our dataset, we would anticipate both features to have similar importance. Let’s verify this by examining the feature importance values from our random forest.
rf.feature_importances_3. Hyperparameters
At this point, you can read and understand almost all the hyperparameters in the documentation. Notice how RandomForests and DecisionTrees share many parameters:
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
You already learned n_estimators which determines the number of trees.
rf = RandomForestClassifier(random_state=0, n_estimators=1000)
rf.fit(X_train, y_train)len(rf.estimators_)Random forests do not evaluate all features at every split. Instead, they randomly select a subset of features and search for the best split within that subset. This subset size is controlled by max_features. The default value is "sqrt", meaning the algorithm will consider √p features at each split, where p is the total number of features. For example, if X has 16 features, each split will be evaluated using only 4 randomly chosen features.
rf = RandomForestClassifier(random_state=0, max_features='sqrt')
rf.fit(X_train, y_train)you can provide integers or fractions as well. See the details in the documentation.
n_jobs controls how many CPU cores the random forest is allowed to use when training or predicting.
A random forest trains many trees independently, so these trees can be built in parallel. That is exactly what n_jobs does:
n_jobs=NoneThis is the default. It means “use 1 core”.n_jobs=1Explicitly use a single CPU core.n_jobs=-1Use all available CPU cores on your machine. This is the typical setting when you want faster training.n_jobs=kUse exactly k cores (for examplen_jobs=4uses 4 cores).
Both .fit() and .predict() will run faster when more cores are used because each tree is handled by a different core.
Therefore n_jobs=-1 can be especially helpful when hyperparameter tuning with large datasets. Let’s create a dataset with 1 million datapoints.
# DUMMY DATA GENERATION
# DO NOT MODIFY
np.random.seed(0) # fixed seed so that everyone gets the same data
n = 1000000
X = np.random.rand(n, 2) * 2
y = np.zeros(n, dtype='int')
y[(X[:, 0] > 1) & (X[:, 1] > 1)] = 1
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)# runs in ~19 seconds on my computer
rf = RandomForestClassifier(random_state=0, n_jobs=1) # n_jobs = 1
rf.fit(X_train, y_train)# runs in ~3 seconds on my computer
rf = RandomForestClassifier(random_state=0, n_jobs=-1) # n_jobs = -1
rf.fit(X_train, y_train)YOUR TURN
We will not cover every hyperparameter in detail. At this point, you know enough to read the documentation and learn the rest through experimentation.
- Open the RandomForestClassifier documentation.
- Read through all the hyperparameters, their descriptions, and the values they can take.
- Choose a few hyperparameters we did not discuss and experiment with them, just like we did above, to see how they affect the forest.
# YOUR CODE HERE