How do neural networks learn?

Nobody writes the rules. Instead of a programmer coding "a cat has pointed ears and whiskers", the network is shown thousands of labelled photos and finds its own patterns. That is the key difference between traditional programming and machine learning.

The four steps of learning

1. Forward pass: an example goes in, each artificial neuron multiplies its inputs by its weights, adds them up and passes the result on, layer by layer, until the network gives an output.
2. Measure the loss: compare the output with the right answer. A common loss for numbers is the squared error, (prediction − target)². The bigger the loss, the worse the guess.
3. Backpropagation: work backwards through the network to find the gradient: how much the loss would change if each weight changed slightly.
4. Update the weights: nudge every weight a small step in the direction that lowers the loss, then repeat with the next example.

The learning rate sets the size of each step. Too small and learning crawls. Too big and the weights overshoot and the loss can bounce around or blow up.

Worked example: one neuron, one weight

Take the simplest possible network: one weight w, where the prediction is w × x. We want it to learn that an input of 2 should give 6, so the right weight is 3. Start with w = 1 and a learning rate of 0.05.

1. Forward pass: prediction = 1 × 2 = 2.
2. Loss: (2 − 6)² = 16.
3. Gradient: for squared error, dL/dw = 2 × (prediction − target) × x = 2 × (2 − 6) × 2 = −16. Negative means increasing w will lower the loss.
4. Update: w = 1 − 0.05 × (−16) = 1 + 0.8 = 1.8.

Now the prediction is 1.8 × 2 = 3.6 and the loss is (3.6 − 6)² = 5.76, down from 16. One more step: the gradient is 2 × (3.6 − 6) × 2 = −9.6, so w becomes 1.8 + 0.48 = 2.28, and the loss falls to about 2.07. Each step gets closer to w = 3, and each step is smaller, because the gradient shrinks as the loss gets near its minimum.

Here x is the weight and y is the loss. Gradient descent is like walking downhill in fog: you can't see the bottom, but you can feel which way the ground slopes and take a step that way. A real network does exactly this, but in a landscape with millions of dimensions, one for every weight. Try updating a weight yourself, then press Mark to check it in our AI Models course.

Backpropagation

In a real network, a weight deep inside affects the output through many layers. Backpropagation uses the chain rule from calculus to pass the error backwards, layer by layer, working out each weight's share of the blame in one efficient sweep. The method was made famous by a 1986 paper in Nature by David Rumelhart, Geoffrey Hinton and Ronald Williams. Hinton later shared the 2024 Nobel Prize in Physics for foundational work that enables machine learning with artificial neural networks.

Epochs, batches and overfitting

• Batches: instead of updating after every single example, networks usually average the gradient over a small batch, which is faster and smoother.
• Epochs: one pass through the whole training set is an epoch. Training often takes many epochs.
• Overfitting: a network can memorise its training examples instead of learning the general pattern. To catch this, some data is held back as a test set the network never trains on. If it does well on training data but badly on the test set, it has overfitted.

Large language models learn the same way, by gradient descent on a loss, with the task of predicting the next token in huge amounts of text. Our guide to tokens in AI explains what they predict. You can watch a network learn in real time in the Neural Network Trainer, and race different learning rates downhill in the Gradient Descent Simulator.

Common mistakes

• Thinking the network stores the examples. It stores weights. The examples shape the weights and are not kept inside.
• Moving the weight the same way as the gradient. Gradient descent moves the opposite way, downhill, which is why there is a minus sign.
• Assuming a bigger learning rate is always faster. Too big and training can overshoot and fail.
• Judging a network only on its training data. Check it on data it has never seen.
• Thinking learning happens while you use it. Most networks learn during training, then their weights are fixed when they are used, which is called inference.

Read next

AI · How AI worksNeural network weights and biases explainedWhat weights and biases are in a neural network, how a neuron uses them, why the bias matters, how to count them, and how training learns them.