Module 2: Data Preparation & Understanding
In Module 1 you trained models on datasets that were ready to use, and in the End-to-End ML Project you went through all stages of a project once. Real data is rarely ready to use. In this module you study the two data stages in depth: preparing the data and exploring it. Then you look at a question that decides whether a model works in practice: does your data look like the data the model will be used on?
Data Preparation comes before Data Understanding in this module, the other way around from CRISP-DM. To understand a dataset, you often have to prepare it first, for example scale features before you compare them. The End-to-End ML Project explains this order in more detail.
What you will be able to do
After this module you can:
- scale, transform, impute, merge and encode data without leaking information from the test data;
- explore the distributions, the relationships and the overall structure (PCA) of a dataset;
- recognize leakage and distribution shift, and split data by group when samples belong together.
Self-study
Work through the activities in order:
- Data Preparation: scaling, transforming, merging and encoding data, and handling missing values.
- Data Understanding: distributions, relationships between variables, and principal component analysis (PCA).
- Leakage & Distribution Shift: what a dataset represents, data unlike the training data, data from different sources, leakage, and how to make your development data match the data a model will be used on.
- Review: a summary of the module and a short quiz.
DataLab
After the Self-study, apply what you learned in the DataLab project: Iteration 2.