Weights and biases in a neural network

In a neural network, weights and biases are the numbers the network learns. Every connection between two neurons has a weight, which says how strongly one neuron's output affects the next. Every neuron also has a bias, a number added on that shifts its output up or down. A neuron multiplies each input by its weight, adds them up, adds the bias, and passes the result through an activation function. Training is the process of finding the weights and biases that make the network's predictions good.

How a neuron uses weights and biases

Google's Machine Learning Crash Course describes each neuron as working like a small linear model: take each input, multiply it by its own weight, add them all up, then add the bias.

The result z is then passed through an activation function, a nonlinear step such as ReLU (which outputs z if it is positive and 0 otherwise) or the sigmoid (which squashes z to between 0 and 1). The activation is what lets a network learn relationships that are not straight lines.

What weights do

• A large positive weight means that input pushes the neuron's output up strongly.
• A negative weight means that input pushes the output down.
• A weight near zero means the input barely matters. Google's glossary notes that a feature with a weight of 0 is effectively removed from a model.

What the bias does

The bias is an offset. Google's glossary compares it to the y-intercept of a line: in y = 0.5x + 2, the bias is 2. Without a bias, a neuron's weighted sum is always 0 when all its inputs are 0, so every line it can draw has to pass through the origin. The bias frees it from that, shifting the point where the activation function switches on. You can think of it as setting how easily the neuron "fires".

Worked example: one neuron

A neuron has two inputs, x₁ = 3 and x₂ = 5, with weights w₁ = 0.8 and w₂ = −0.5, and a bias b = 1.2. It uses ReLU. Find its output.

1. Weighted sum: 0.8 × 3 + (−0.5) × 5 = 2.4 − 2.5 = −0.1.
2. Add the bias: z = −0.1 + 1.2 = 1.1.
3. ReLU: z is positive, so the output is 1.1. (With a sigmoid instead, the output would be 1 ÷ (1 + e⁻¹·¹) = 0.750.)

Now change only the bias to b = −0.6. Then z = −0.1 − 0.6 = −0.7, and ReLU outputs 0: the same inputs and weights, but the neuron no longer fires. That is the bias at work.

Practice computing a weighted sum in the AI Models course.

How many weights and biases does a network have?

Weights and biases together are the network's parameters. In a fully connected network, every neuron in one layer connects to every neuron in the next:

• Weights between two layers = (neurons in the first) × (neurons in the second).
• Biases = one for every neuron that is not an input.

Google's crash course works through the same kind of count: each hidden neuron fed by three inputs needs 4 parameters, 3 weights and a bias. Large language models have billions of parameters, but each one is still just a weight or a bias.

How weights and biases are learned

A network starts with its weights and biases set to small random values, so its first predictions are poor. Training then repeats a loop:

1. Make predictions for some training examples.
2. Measure how wrong they are with a loss function.
3. Work out, for every weight and bias, which way to nudge it to reduce the loss (backpropagation).
4. Nudge them all a little (gradient descent), and repeat.

The how do neural networks learn guide walks through this step by step, and you can watch the weights change live in the Neural Network Trainer. Too many parameters for the amount of data can lead to overfitting.

Common mistakes

• Thinking the bias is a weight on an input. It is added once per neuron, not multiplied by an input (though it can be treated as a weight on a constant input of 1).
• Forgetting biases when counting parameters. Add one per non-input neuron.
• Confusing this bias with bias in data. The bias term is just an offset; a biased dataset or model is a different idea.
• Thinking weights are set by hand. They start random and are learned during training.

Read next

AI · How AI worksNeural network visualization: watch a network learnBuild a small neural network and watch it learn live: its decision boundary, weights and loss curve. A neural network playground with experiments to try.