Skip to main content

Module 5: Tree-Based Algorithms

Decision trees, and models that combine many of them, are among the strongest models for tabular data. In this module you see how a single tree learns, and how random forests and gradient boosting combine many trees into a better model.

What you will be able to do

After this module you can:

  • explain how a decision tree chooses its splits and why it overfits;
  • explain how random forests and gradient boosting combine trees, and tune their main hyperparameters;
  • interpret feature importances with care;
  • use a gradient boosting library with early stopping on a validation set.

Self-study

Work through the activities in order:

  1. Decision Trees: a decision tree by hand and with scikit-learn, how a tree learns, overfitting, hyperparameters, and regression trees.
  2. Random Forests: many trees trained on random parts of the data, their hyperparameters, voting and extra-trees.
  3. Gradient Boosting: trees that each correct the errors of the previous ones, their hyperparameters, XGBoost and LightGBM, AdaBoost and stacking.
  4. Review: a summary of the module and a short quiz.

DataLab

After the Self-study, apply what you learned in the DataLab project: Iteration 5.