How a Neuron Makes a Decision
See how a neuron turns its total into the signal it passes on, and why that last bend matters so much.
Last lesson’s beach neuron had a blunt rule: total above 0, go; below 0, stay home. A total of 0.1 and a total of 50 got the exact same answer. It couldn’t tell “meh, why not” from “sun’s out, everyone’s coming, grab the towels”.
Real neurons add one more step. They pass the total through an activation function: a curve that bends the total into the signal the neuron sends on. Three you’ll meet everywhere:
Step
A hard yes/no switch. Sounds sensible, but it’s nearly useless for learning: nudge a weight a little and the answer almost never changes, so the network can’t tell which way is better.
Sigmoid
Squeezes any total into 0 to 1. Reads like confidence: 0.5 means “no idea”, 0.99 means “absolutely”.
ReLU
Negatives become 0, positives pass straight through. Almost embarrassingly simple, and it, or a close cousin, powers most modern networks.
There’s a deeper reason for the bend. Without it, stacking neurons is pointless: a chain of multiply-and-add steps collapses into one big multiply-and-add, a single straight line no matter how many layers you pile up. The bend is what lets layers build curves, as the next lesson shows.
Key takeaways
- After adding everything up, a neuron passes its total through an activation function: a bend, or “squish”.
- A step function gives a hard yes/no, but small changes to the weights almost never change it, so learning gets no hints.
- sigmoid turns the total into a 0-to-1 confidence; ReLU zeroes out negatives and lets positives through.
- Without a bend, any stack of layers collapses into one straight line. The bend is what makes depth worth having.
Check your understanding
Go deeper (more technical)
In symbols: output = f(w₁x₁ + … + wₙxₙ + b), where f is the activation function. Sigmoid is f(z) = 1 / (1 + e⁻ᶻ); ReLU is f(z) = max(0, z).
Why the step fails: learning needs the slope, how much the output changes when a weight changes a tiny bit. The step function is flat everywhere except at the jump, so its slope is zero almost everywhere. Sigmoid has a slope everywhere, which made it the standard choice for decades. Its weakness: for big totals it flattens out, the slope shrinks towards zero, and deep stacks of sigmoids learn painfully slowly (the “vanishing gradient” problem). ReLU keeps a slope of 1 for every positive total, a big reason deep networks became trainable around 2010–2012.
Why no bend means no depth: if layer 1 computes 2x + 1 and layer 2 computes 3 × (that) − 4, the whole thing is just 6x − 1. Any stack of multiply-and-add layers equals one multiply-and-add layer.