Skip to main content

Module 2: Data Preparation & Understanding

In Module 1 you trained models on datasets that were ready to use, and in the End-to-End ML Project you went through all stages of a project once. Real data is rarely ready to use. In this module you study the two data stages in depth: preparing the data and exploring it. Then you look at a question that decides whether a model works in practice: does your data look like the data the model will be used on?

Data Preparation comes before Data Understanding in this module, the other way around from CRISP-DM. To understand a dataset, you often have to prepare it first, for example scale features before you compare them. The End-to-End ML Project explains this order in more detail.

What you will be able to do

After this module you can:

  • scale, transform, impute, merge and encode data without leaking information from the test data;
  • explore the distributions, the relationships and the overall structure (PCA) of a dataset;
  • recognize leakage and distribution shift, and split data by group when samples belong together.

Self-study

Work through the activities in order:

  1. Data Preparation: scaling, transforming, merging and encoding data, and handling missing values.
  2. Data Understanding: distributions, relationships between variables, and principal component analysis (PCA).
  3. Leakage & Distribution Shift: what a dataset represents, data unlike the training data, data from different sources, leakage, and how to make your development data match the data a model will be used on.
  4. Review: a summary of the module and a short quiz.

DataLab

After the Self-study, apply what you learned in the DataLab project: Iteration 2.