Gradient Descent: Finding Better Weights
Understand how a network finds good weights by itself: measure the mistakes, then nudge the weights downhill.
Setting weights by hand works for 4 numbers, not for millions. So how does a network learn? With a simple loop, repeated over and over: guess, measure the mistake, nudge the weights, repeat.
You already know how to score the guesses: the loss, one number for how wrong the network is overall. Now picture every possible weight as a spot on a hillside, with the loss as its height. The lowest point of the valley is the best weight.
The gradient tells you which way is downhill from where you stand. So the network takes a small step that way, measures again, and steps again. That is gradient descent. How big each step is, the step size, matters: too small is slow, too big overshoots and can blow up.
Key takeaways
- The loss (mistake score) measures how wrong the model is: lower is better.
- The gradient says which way to change each weight to make the loss smaller.
- Gradient descent: take a small step downhill, measure again, repeat thousands of times.
- The step size (learning rate) must be just right: too small is slow, too big blows up.
Check your understanding
Go deeper (more technical)
The mistake score here is the mean squared error: the average of (guess − right answer)². The gradient is its slope with respect to the weight, which calculus gives directly. Each step does weight ← weight − step size × gradient.
A real network has millions or billions of weights, so the "valley" has millions of directions. A method called backpropagation works out the gradient for every weight at once, starting at the output and going back layer by layer. And instead of using all examples per step, training uses a small random handful each time ("stochastic" gradient descent), which is much faster.