Skip to main content

Module 3 Review

You have learned to evaluate and compare models, and you have looked inside three of them. This page sums up the Self-study of this module and checks the main ideas.

Summary

  • Modeling & Evaluation: cross-validation for a reliable estimate (stratified for classification, by group when samples belong together); a validation set or cross-validation for every choice, and the test set once at the end; feature selection, model selection and hyperparameter tuning; underfitting and overfitting from the training and validation errors; metrics such as precision, recall, (macro-)F1, MAE, RMSE and R².
  • Linear Regression: a cost, gradient descent, several features, polynomial features, and regularization with Ridge and Lasso.
  • Logistic Regression: from a line to a probability, the decision threshold, precision-recall and ROC curves, and regularization with C.
  • Clustering: distances and scaling, K-Means, the elbow method and the silhouette score, and density-based clustering.

Quiz

These questions are easy if you have worked through the activities. Choose an answer to see whether it is right and why.

A model has a training error close to 0 and a much higher validation error. What is the most likely diagnosis?

  • Underfitting: the model is too simple for the data.
  • Overfitting: the model follows the noise in its training data.
  • A bug in the metric, because the training and the validation error should be about equal.

You compare ten values of a hyperparameter. On which data do you compare them?

  • On the test set, because it gives the most honest estimate of the performance on new data.
  • On the training data, because the model was fitted on it.
  • On a validation set, or with cross-validation.

95% of the samples are negative, and a model predicts “negative” for every sample. Which metric shows the problem?

  • Accuracy, which is 95%, because it counts every correct prediction.
  • The F1 score of the positive class, or the macro-F1.
  • The mean absolute error of the predictions, computed on the test set.

In LogisticRegression, what does a larger value of C do?

  • It weakens the regularization.
  • It strengthens the regularization, so the model becomes simpler and smoother.
  • It moves the decision threshold away from 0.5.

Why do you scale the features before K-Means?

  • Because K-Means only accepts values between 0 and 1, like a probability.
  • Because K-Means uses distances, which large numbers dominate.
  • Because scaling sets the number of clusters that K-Means looks for.