import numpy as np
import pandas as pd
import matplotlib.pyplot as pltData Understanding
Data understanding is the phase where you explore, describe, and verify your data before modeling. It’s about knowing what you’re working with, the data types, distributions, relationships, missing values, and potential issues. Good data understanding ensures your later analysis is meaningful and reliable.
You should already know the basics of data understanding (see the prerequisites on the course homepage), such as checking missing values, describe and info methods etc. If you need a refresher, the pandas getting started tutorials cover them. In this notebook we will cover intermediate concepts.
1. Understanding the Distribution of Each Variable
A distribution shows how values in your dataset are spread or concentrated. It reveals patterns such as skewness, outliers, and common ranges, helping you understand the typical and extreme cases. Knowing the distribution of each variable is essential because it influences which models, transformations, and evaluation methods are appropriate.
from sklearn.datasets import fetch_california_housing
X, y = fetch_california_housing(as_frame=True, return_X_y=True)We use histograms to infer the distribution of the data. Histograms are graphical representations that group data into bins and show the frequency of values within each bin. pandas makes it very easy to plot histograms.
X.hist()
plt.tight_layout()you could also use matplotlib:
plt.hist(X['HouseAge'])the default number of bins in matplotlib is 10. You can adjust it and as you can see it might suddenly give a different view:
plt.hist(X['HouseAge'], bins=50)Two very important points:
- The number of bins you choose can dramatically change how the dataset appears.
- We don’t plot distributions just for fun. As a competent data scientist, you should be alerted by unusual patterns such as the sudden spike after 50 and investigate their cause.
2. Exploring Relationships Between Variables
The distribution of a variable tells us about that variable alone. However, in machine learning, we work with multiple features and a target, and understanding the relationships between them is key. If two features are highly related, for instance, one column shows length in centimeters and another shows length in inches, they contain the same information. Keeping both would be redundant and could confuse the model, so one should be dropped. On the other hand, if a feature shows a strong relationship with the target variable, that’s a sign it may be useful for prediction. By exploring these relationships, we can identify which features provide real value and which ones add noise or duplication.
The most intuitive way to explore the relationship between two variables is to plot them together, placing one variable on the x-axis and the other on the y-axis.
plt.scatter(X['MedInc'], y)
plt.xlabel('Median Income')
plt.ylabel('Median House Value')Looking at this plot, we can see that blocks with higher median income tend to have higher median house values. Alternatively, we can say that as the median house value of a block increases, there is a tendency for the income to rise as well. This relationship makes sense because families with higher incomes can typically afford more expensive homes, leading to this positive association.
However, we cannot say that one causes the other, because correlation does not imply causation. Both variables may be influenced by other underlying factors such as location, proximity to job opportunities, or neighborhood quality, which can independently affect both income and house value.
Please read the following Wiki: https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation
The relationship above shows how a feature relates to the target variable, and in general, we prefer features that have a clear connection with the target. However, the absence of an obvious relationship does not necessarily mean that a feature is useless, it might still contribute valuable information when combined with others.
Now, let’s look at a relationship between two features:
plt.scatter(X['AveRooms'], X['AveBedrms'])
plt.xlabel('AveRooms')
plt.ylabel('AveBedrms')This relationship also makes sense: if a house has more rooms, we would naturally expect it to have more bedrooms, and vice versa. However, for modeling purposes, using both features is not ideal, since they contain overlapping information.
We can also quantify the strength of relationships between variables, with the most common measure being correlation. However, correlation captures only linear relationships, while non-linear relationships can still hold valuable information for machine learning models. In pandas it is very easy to check the correlation between all the columns as follows:
X.corr()because X does not contain y, we only see the feature-feature correlations. We can create a new df that combines X and y to see all correlations:
df = X.copy()
df['target'] = y
df.corr()We can see that the correlation between AveBedrms and AveRooms is 0.85, and between MedInc and the target is 0.69, both quite high, as expected. This numerical approach allows us to quickly identify and filter strong correlations without having to plot each pair. For example, we can find the feature most strongly correlated with the target as follows:
df.corr()['target'].sort_values(ascending=False)3. Principal Component Analysis (PCA)
We have explored relationships between pairs of variables and examined their distributions. But is there a way to look at all the data at once? Each data point here is described by eight features, and since we cannot visualize an eight-dimensional plot, we are limited to viewing only two dimensions at a time (3D at best). PCA helps overcome this limitation by projecting high-dimensional data into two or three dimensions that capture most of the variation, allowing us to see the overall structure and patterns in the dataset. It reduces the dimensions from eight to two hence the name dimensionality reduction.
We can perform PCA easily using scikit-learn. For now, ignore the code and focus on understanding the output.
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
# 1) Scale features (important for PCA)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# 2) Fit PCA to 2 components
pca = PCA(n_components=2, random_state=0)
X_pca = pca.fit_transform(X_scaled)
# 3) Put into a DataFrame (optional)
pca_df = pd.DataFrame(X_pca, columns=['PC1', 'PC2'], index=X.index)
# 4) Plot the two components
plt.figure()
plt.scatter(pca_df['PC1'], pca_df['PC2'], s=12, alpha=0.6)
plt.xlabel(f"PC1")
plt.ylabel(f"PC2")
plt.title("PCA projection to 2 components")This plot shows the result of projecting the California housing data onto two principal components using PCA. Each point represents one district from the dataset, summarized by eight original features and reduced to two dimensions (PC1 and PC2).
We can see that most data points cluster near the origin, meaning most districts have similar overall feature patterns. A few points lie far from the cluster, indicating districts that differ strongly from the average, these could be outliers, for example very high-income or high-value areas.
In machine learning, an important use of PCA is to examine the distribution of the data source. By projecting both the training and test sets onto the principal components, we can visually check whether they come from similar distributions. For example, if your training data contains black and orange cats while your test data contains only white cats, PCA would reveal that the test samples form a separate cluster. This indicates a distribution shift, meaning the model may not generalize well because the test data differ significantly from what it has seen during training.
Now let’s look at the code:
# why did we scale X?
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)PCA is sensitive to the scale of features because it relies on variance. If one feature (like income) has much larger values than another (like number of bedrooms), it would dominate the principal components. StandardScaler fixes this by transforming every feature to have mean = 0 and standard deviation = 1, so all features contribute equally to PCA.
# fit_transform was discussed in the previous notebook
# n_components is the important parameter here
# random state for repeatable results
pca = PCA(n_components=2, random_state=0)
X_pca = pca.fit_transform(X_scaled)Here we apply PCA to the scaled data and ask for two principal components (n_components=2). PCA finds the two directions (linear combinations of the original features) that capture the most variance in the data. X_pca is the new dataset with only two columns, these are the coordinates of each data point in the new 2D space that summarizes most of the original information.
PCA works by finding new axes (principal components) that are linear combinations of the original features. As a result, PCA cannot represent nonlinear patterns in the data.
Therefore there are other dimensionality reduction techniques worth exploring. Two popular nonlinear methods are:
- t-SNE (available in scikit-learn): t-SNE documentation
- UMAP (available through its own library): UMAP documentation
These techniques can capture complex, nonlinear relationships in the data that PCA might miss. They are valuable tools to visualize high-dimensional data and to explore clusters or patterns beyond what PCA can reveal.
4. Exercise
YOUR TURN
Plot the breast cancer dataset using PCA reduced to 2 dimensions, and color the data points according to their class labels (y). Add axis labels, a title, and a legend. Then reflect on what you see, do the two classes separate clearly in this 2D projection?
Important: do not use y when performing PCA, apply PCA only to X like we have done above. After the transformation, color the points by their class labels (y) when plotting.
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(as_frame=True, return_X_y=True)X.head()