Skip to main content

Module 5 Review

You have seen how decision trees learn, and how random forests and gradient boosting combine many trees. This page sums up the Self-study of this module and checks the main ideas.

Summary

  • Decision Trees: a tree splits the data on one feature and one threshold at a time, choosing the split that reduces the impurity most (Gini). Without limits it grows until every leaf is pure and overfits; max_depth, min_samples_split and min_samples_leaf limit its growth. Trees also work for regression.
  • Random Forests: many trees, each trained on a random sample of the data with random features at each split; averaging them reduces the variance. Feature importances show what the model uses, not what causes the outcome; the permutation importance is a more reliable check.
  • Gradient Boosting: trees trained one after the other, each on the remaining errors of the ones before. The learning rate and the number of trees are tuned together, with early stopping on a validation set. XGBoost, LightGBM and CatBoost are fast implementations; AdaBoost and stacking are other ways to combine models.

Quiz

These questions are easy if you have worked through the activities. Choose an answer to see whether it is right and why.

A decision tree without limits reaches 100% training accuracy on noisy data. Why?

  • It splits until every leaf is pure, noise included.
  • Noisy data is easy to learn for a decision tree, because trees ignore small errors.
  • The Gini impurity ignores the noisy samples when it chooses the splits.

Why does a random forest usually predict better than one deep tree?

  • Its trees are shallower, so they cannot overfit.
  • Every tree in the forest sees all the training data and all the features, so it learns more.
  • It averages many different trees, so their errors partly cancel.

In gradient boosting, what does each new tree learn?

  • The original targets, independently of the trees before it.
  • The remaining errors of the trees before it.
  • Which features are the most important.

You lower the learning rate of gradient boosting from 0.3 to 0.03. What else should you change?

  • Use more trees, or early stopping.
  • Nothing: the learning rate does not interact with other settings.
  • Use deeper trees, so that each tree can make up for the smaller steps.