Where Does AI Get Its Training Data?
Know where the answers in training data come from, and why their quality caps how good a model can be.
Supervised learning needs examples with answers, called labels. Somebody, or something, has to supply them. There are four main sources:
- People, paid to label. Armies of workers draw boxes around pedestrians, transcribe audio and rate chatbot replies. Slow, expensive, and people disagree more than you’d think. Is a hot dog a sandwich?
- You, without noticing. The spam button, a skipped song, a thumbs-down. Each one is a free label.
- The data itself. Hide the next word of a sentence and you have a question and its answer, no human needed. This trick, self-supervised learning, lets chatbots learn from trillions of words.
- The AI’s own experience. A game-playing AI plays millions of games against itself. Who won is the label.
Whatever the source, the model swallows it whole. Wrong labels teach wrong lessons, and whatever is missing from the data is missing from the model. Programmers have a blunt name for this: garbage in, garbage out.
Key takeaways
- Labels are the right answers in training data. Someone, or something, has to supply them.
- Paid labellers are careful but slow and expensive. Clicks from users are free but noisy.
- Text can label itself: hide the next word and the sentence holds the answer. That’s how chatbots learn at scale.
- A model can’t be better than its data: wrong labels and missing cases turn into wrong answers.
Check your understanding
Go deeper (more technical)
Human labelling is a real industry. Modern chatbots rely on people writing example answers and ranking replies. Because people disagree, careful datasets have several people label each example and keep the majority answer, or flag the examples they argue about.
ImageNet, the photo dataset that kicked off the deep learning boom, was labelled through Amazon Mechanical Turk, a crowd-work website: tens of thousands of workers from over 160 countries sorted about 14 million images.
Labels from user behaviour are called implicit feedback. They’re plentiful but slippery: you skipped that song because you were driving, not because you hated it. Self-supervised learning sidesteps labellers entirely, which is why it scales to the whole internet. But the internet has its own errors and biases, and the model learns those too.