Backpropagation: How Neural Networks Know What to Change
See how a network works out, for every weight at once, how much it was to blame for a mistake.
Gradient descent says: nudge every weight a little bit downhill. Easy with one weight. A real network has billions, buried layers deep. When the final answer comes out wrong, which ones should change, and by how much?
Think of a restaurant. A customer sends back a cold soup. The waiter didn’t cook it. The chef did, on a stove the manager never fixed. The complaint travels backwards through the kitchen, and everyone passes some of it on, in proportion to their part in the disaster.
backpropagation does that with numbers. Start at the output: how far off was the answer, and in which direction? That’s the error signal. Then walk backwards:
- A weight’s blame is the error signal arriving at it times the value that flowed through it on the way forward. A connection that carried nothing gets no blame.
- A neuron’s error signal is the signal after it times the weight in between. A neuron with a big outgoing weight gets a big share of the blame.
One sweep from the output back to the input hands every weight its nudge, for about twice the work of the forward pass, whether the network has ten weights or ten billion. Without that trick, training modern AI would be hopeless.
Key takeaways
- Backpropagation answers “how much did each weight contribute to the mistake?” for every weight at once.
- It starts from the error signal at the output and passes blame backwards, layer by layer.
- A weight’s blame = the error signal arriving at it × the value that flowed through it. Connections that carried more get more.
- One backward pass costs about twice a forward pass, so it scales to billions of weights. Gradient descent then uses the blame to nudge.
Check your understanding
Go deeper (more technical)
Under the hood, backpropagation is the chain rule from calculus applied over and over. If y depends on h, and h depends on w, then how much y moves when w moves = (how much y moves per unit of h) × (how much h moves per unit of w). Every layer multiplies one more factor onto the chain.
The simulation scores its miss as half the squared error, which makes the output’s error signal simply guess − answer, and its neurons skip the squish to keep the arithmetic friendly. With squish functions, each neuron’s error signal is also multiplied by the slope of its squish at its current total. That’s where step functions fail (slope 0) and where deep stacks of sigmoids struggle (slopes of at most 0.25, multiplied over and over).
The idea was discovered several times, but a 1986 paper by Rumelhart, Hinton and Williams made it famous. Today, tools like PyTorch do it automatically: you write the forward pass and “autograd” builds the backward pass for you.