How LLMs Learn From the Internet and Other Data
See where an LLM’s knowledge comes from: what pre-training data is, how it’s cleaned, and what it produces.
A model’s knowledge has to come from somewhere, and for pre-training that somewhere is mostly the public web, plus books, code, scientific papers and Wikipedia. Llama 3 was trained on about 15 trillion tokens. Reading non-stop, day and night, a person would need something like 80,000 years to get through it.
The raw web, though, is mostly junk: cookie banners, menus, spam, gibberish, and the same article copied a thousand times. Feed that in unfiltered and the model gets very good at writing cookie banners. So before training, the data goes through a long cleaning pipeline:
Pull out the main text, drop pages in unwanted languages or with too little real content, score quality (often with a small model trained to spot well-written pages), remove near-duplicates, and scrub personal data such as phone numbers. Teams then choose the mix: how much web, how much code, how many books. Code, for instance, appears to help with reasoning, not only with programming.
Then comes the expensive part. The model reads through the mix, predicting each next token and nudging its weights after every batch, for weeks to months on thousands of GPUs: the largest Llama 3 model took around 30 million GPU-hours. The result is a base model: superb at continuing text, with a broad, blurry memory of everything it read, but not yet an assistant. Ask it a question and it may answer with three more questions, because that’s how plenty of web pages continue.
Two side effects are worth remembering. The model inherits the biases and errors of its data. And its knowledge stops at a cutoff date: anything that happened after the data was collected simply isn’t in there.
Key takeaways
- Pre-training data is mostly filtered web text, plus code, books, papers and Wikipedia: trillions of tokens.
- Cleaning matters: extract the text, filter quality and language, remove duplicates, scrub personal data.
- Pre-training is next-token prediction over that data, for weeks to months on thousands of GPUs.
- The result is a base model: knowledgeable but not yet helpful, with a knowledge cutoff date.
Check your understanding
Go deeper (more technical)
Hugging Face’s open FineWeb dataset (2024) shows a real pipeline: 96 Common Crawl snapshots, text extraction, language and quality filters and de-duplication, ending at about 15 trillion tokens. Its FineWeb-Edu variant used Llama 3 70B to rate pages for educational value, then trained a small classifier to apply those ratings at scale.
Most large models see their data roughly once, not over and over. Teams run small “ablation” models to decide the mix before committing. Synthetic data, text written by other models, is a growing ingredient. And who may use which text for training is the subject of ongoing legal and ethical debate.