Module 3 Review
You have learned to evaluate and compare models, and you have looked inside three of them. This page sums up the Self-study of this module and checks the main ideas.
Summary
- Modeling & Evaluation: cross-validation for a reliable estimate (stratified for classification, by group when samples belong together); a validation set or cross-validation for every choice, and the test set once at the end; feature selection, model selection and hyperparameter tuning; underfitting and overfitting from the training and validation errors; metrics such as precision, recall, (macro-)F1, MAE, RMSE and R².
- Linear Regression: a cost, gradient descent, several features, polynomial features, and regularization with Ridge and Lasso.
- Logistic Regression: from a line to a probability, the decision threshold, precision-recall and ROC curves, and regularization with
C. - Clustering: distances and scaling, K-Means, the elbow method and the silhouette score, and density-based clustering.
Quiz
These questions are easy if you have worked through the activities. Choose an answer to see whether it is right and why.
A model has a training error close to 0 and a much higher validation error. What is the most likely diagnosis?
- Underfitting: the model is too simple for the data. Underfitting shows as a high training error. Here the training error is close to 0.
- Overfitting: the model follows the noise in its training data. Right. A large gap between training and validation error is the sign.
- A bug in the metric, because the training and the validation error should be about equal. The two errors are rarely equal: the training error is usually lower.
You compare ten values of a hyperparameter. On which data do you compare them?
- On the test set, because it gives the most honest estimate of the performance on new data. Once you choose on the test set, its score is no longer honest.
- On the training data, because the model was fitted on it. The training error keeps going down for more flexible settings, so it would always pick the most flexible one.
- On a validation set, or with cross-validation. Right. The test set stays unseen until the final check.
95% of the samples are negative, and a model predicts “negative” for every sample. Which metric shows the problem?
- Accuracy, which is 95%, because it counts every correct prediction. 95% looks good, and that is the problem: accuracy hides it.
- The F1 score of the positive class, or the macro-F1. Right. The F1 of the positive class is 0, so the macro-F1 is low too.
- The mean absolute error of the predictions, computed on the test set. The MAE is a regression metric.
In LogisticRegression, what does a larger value of C do?
- It weakens the regularization. Right.
Cworks in the inverse direction of the regularization strength. - It strengthens the regularization, so the model becomes simpler and smoother. The other way around: a smaller
Cmeans stronger regularization. - It moves the decision threshold away from 0.5. The threshold is a separate choice, made after training.
Why do you scale the features before K-Means?
- Because K-Means only accepts values between 0 and 1, like a probability. K-Means accepts any numbers. The problem is what the distances mean.
- Because K-Means uses distances, which large numbers dominate. Right. Scaling gives every feature a comparable influence.
- Because scaling sets the number of clusters that K-Means looks for. You choose the number of clusters yourself, for example with the elbow method or the silhouette score.