Overfitting vs underfitting

Overfitting vs underfitting is the question of whether a machine learning model has learned too much or too little from its training data. An overfit model memorizes the training examples so closely, noise and all, that it predicts badly on new data. An underfit model is too simple to capture the pattern, so it does badly even on the data it trained on. The goal is a model in between that generalizes: it makes good predictions on data it has never seen.

Overfitting vs underfitting at a glance

Each fit tries to describe the same seven points, which follow a U shape with some random noise. The flat line ignores the U entirely. The smooth U follows the real pattern. The pink curve passes through every single point, which looks perfect on the training data, but it has learned the noise: between and beyond the points, its wiggles predict nonsense.

What is overfitting?

Google's Machine Learning Crash Course describes an overfit model as one that matches, or memorizes, the training set so closely that it fails to make correct predictions on new data. It names two common causes:

• The model is too complex for the amount of data: it has enough parameters to fit every quirk.
• The training set doesn't represent real life, so patterns that hold in the training data don't hold outside it.

A student who memorizes last year's test answers without understanding them is overfitting: a perfect score on the practice test, then failure on a new one.

What is underfitting?

An underfit model doesn't even make good predictions on its training data. It is usually too simple for the problem, like fitting a straight line to a curved pattern, or it has not been trained for long enough. Because it misses the real pattern, it does badly on new data too.

How to tell them apart: loss curves

Split your data into a training set, which the model learns from, and a validation set it never trains on. Plot the loss (error) on both as training goes on:

• Underfitting: both losses stay high. The model can't even fit the training data.
• Good fit: both losses fall and level off close together, with similar shapes.
• Overfitting: the curves diverge. Training loss keeps falling, but after a certain number of steps validation loss starts to rise.

Practice reading loss curves in this question from the AI Models course.

How to fix overfitting

1. Get more, and more representative, training data. The model then has to learn the general pattern because it can't memorize everything.
2. Use a simpler model. Google points to Occam's razor: simpler models usually do better on the test set, which is what matters.
3. Regularization. Add a penalty for complexity to the loss. L2 regularization pushes weights toward zero; a higher regularization rate (λ) lowers the chance of overfitting.
4. Early stopping. Stop training before the model fully converges, around the point where validation loss is lowest. It usually raises training loss slightly but can lower test loss.
5. Dropout (for neural networks). Randomly switch off some neurons during training so the network can't rely on any one path; a higher dropout rate means stronger regularization.

How to fix underfitting

An underfit model is too simple to capture the pattern, so the fixes go the other way:

• Use a more flexible model, for example more neurons or layers in a neural network, or a curve instead of a line.
• Train for longer if the training loss is still falling.
• Reduce regularization. Google notes that too high a regularization rate gives weak weights and poor predictions.
• Give the model better inputs (features) that actually relate to what you are predicting.

Overfitting and underfitting pull in opposite directions, and this balance is often called the bias-variance tradeoff: too simple and the model is biased toward the wrong shape; too complex and its predictions vary wildly with each new training sample.

Worked example: choosing a model

Polynomials of degree 1, 2 and 6 are fitted to the seven training points above by least squares. Six new points from the same U-shaped pattern are kept back as a test set. The mean squared errors are:

The degree 6 model has the lowest training error of all, but the worst test error. Judging a model by its training error alone would pick exactly the wrong one. That is why you always keep data back for testing.

Try it yourself

In the Neural Network Trainer, pick a hard dataset, give the network several large layers and let it run. Watch the training and test loss curves: when the test loss turns upward while the training loss keeps falling, you are watching overfitting happen. Then shrink the network and compare. The how do neural networks learn guide explains the training itself.

Common mistakes

• Judging a model by training accuracy. Only performance on data the model has not seen tells you how it will do in use.
• Tuning on the test set. If you keep adjusting the model to do well on the test set, you start overfitting to it. Use a separate validation set for tuning.
• Assuming bigger is always better. Extra capacity helps an underfit model but makes overfitting more likely.
• Regularizing an underfit model. It makes underfitting worse; add capacity instead.
• Training and test data from different sources. If they don't share the same distribution, the test score misleads you either way.

Read next

AI · How AI worksNeural network visualization: watch a network learnBuild a small neural network and watch it learn live: its decision boundary, weights and loss curve. A neural network playground with experiments to try.