Gradient descent visualization

A gradient descent visualization shows how a machine learning model finds the lowest point of its loss: it looks at the slope where it stands and takes a step downhill, again and again. Our Gradient Descent Simulator races four optimisers (optimizers in the US), SGD, momentum, RMSProp and Adam, from the same starting point down a loss surface drawn as a contour map, with a live loss curve for each.

It opens on a long, narrow ravine with all four optimisers racing at a learning rate of 0.2.

Try it

Gradient Descent Simulator

Race optimisers down a loss landscape and tune the learning rate

Open the simulatorGo to its page

How to use the gradient descent simulator

1. Under Surface, choose a Loss surface: Round bowl, Long ravine, Saddle, Rosenbrock valley, Four minima or Bumpy bowl.
2. Tap the map to choose where the optimisers start.
3. Under Optimisers, turn SGD, Momentum, RMSProp and Adam on or off.
4. Set the Learning rate, from 0.001 to 1, and the Gradient noise, which adds random error to each gradient as a small batch of training data would.
5. Press play under Run and watch the paths on the contour map and the loss curves underneath.

An optimiser that stops moving has settled; one that flies off the map has escaped because its steps grew too large. Practice and help sets questions on single gradient descent steps and on momentum, with worked solutions.

How gradient descent works

The gradient points uphill, in the direction the loss grows fastest. Gradient descent steps the other way, by an amount set by the learning rate η:

Example 1: one step by hand

Take f(x) = x², whose gradient is 2x, starting at x = 3 with a learning rate of 0.1. One step goes to x = 3 − 0.1 × 2 × 3 = 2.4, closer to the minimum at 0. Each step here multiplies x by 0.8, so it closes in steadily.

Experiments to try

Example 2: the ravine

The opening surface, 0.1x² + 4y², is 40 times steeper across than along. At a learning rate of 0.2, the simulator's race ends like this:

SGD zigzags across the steep walls while creeping slowly along the gentle floor. RMSProp evens out the two directions, so it heads almost straight for the bottom.

Example 3: a learning rate that is too high

Raise the learning rate to 0.3 on the ravine. In the steep direction, each SGD step multiplies y by 1 − 8 × 0.3 = −1.4, so it overshoots by more each time: SGD escapes after 9 steps. The other three still settle. Too small a learning rate has the opposite problem: steps so tiny that training crawls.

Check your reading of loss curves in the course, then press Mark:

Example 4: where you start matters

Four minima is Himmelblau's function, with four equally low points. From the opening start, SGD and momentum settle at (3.58, −1.85), while RMSProp and Adam settle at (3, 2), a different minimum. Tap the map near (−1, 1) and all four reach (−2.81, 3.13). Real models have many such valleys, and the starting point helps decide which one training finds.

Example 5: a saddle

On the Saddle, x² − y², the slope is almost flat near the middle. Starting just off the ridge, the optimisers slide towards the centre, then roll off down the y direction. There is no minimum to find: the loss keeps falling and they leave the map.

What the simulator does not show

• Two parameters only. Real networks have millions, so their loss surfaces cannot be drawn. The same update rules apply.
• Set surfaces. You choose from six built-in functions; you cannot type your own.
• No real training data. Gradient noise imitates mini-batches. To train a small network on data, try the Neural Network Trainer.

Common mistakes

• Stepping up the gradient. The gradient points uphill, so gradient descent subtracts it.
• Thinking a bigger learning rate is always faster. Past a limit, steps overshoot and the loss grows.
• Expecting Adam to win every time. On the round bowl, plain SGD settles first in the simulator. The best optimiser depends on the surface.
• Thinking a settled optimiser found the best answer. It found a low point near where it started, which may not be the lowest one.

Learn more

The AI Models course builds from loss and gradient descent to the full training loop, and our guide to how neural networks learn explains backpropagation, which works out the gradients that gradient descent uses.

Read next

AI · How AI worksHow do neural networks learn? Gradient descent explainedHow neural networks learn from examples: weights, the loss, gradient descent and backpropagation, with a worked example of one learning step.