Why AI Hallucinates
Understand why LLMs state false things confidently, and what actually helps.
Ask a chatbot for the biography of an obscure 19th-century painter, and it may hand you a fluent, detailed, completely invented life story, with dates, exhibitions and a tragic ending. That’s a hallucination: plausible-sounding output that isn’t true. It isn’t a glitch on a bad day. It falls straight out of how these models work, for three reasons.
1. The job is “plausible”, not “true”. A model is trained to produce likely continuations. Usually the likeliest continuation is also the true one, because true statements dominate the training data. But when the model doesn’t know, it still produces the most plausible-sounding text, and invented details sound very plausible indeed.
2. Memory is blurry for rare facts. As Track 1 showed, a model keeps patterns, not records. A fact repeated a million times is sharp; one seen once or twice is a faint smear. Ask about it and the probabilities spread over several candidates, but sampling still picks one, in the same confident tone as everything else.
3. Guessing gets rewarded. Many training signals and benchmarks score answers as simply right or wrong. “I don’t know” earns nothing, while a guess sometimes scores. A model optimised against that kind of grading learns to guess, like a student facing a multiple-choice exam with no penalty for wrong answers. A 2025 paper from OpenAI argued this is a major reason hallucinations persist.
What helps: rewarding honest uncertainty in training and evaluation; giving the model trusted sources to read and cite (RAG, Track 6); letting it check facts with tools such as search. And for you, one habit: treat specific claims, such as names, numbers, quotes and references, as leads to verify, not facts to repeat.
Key takeaways
- Hallucinations are fluent, confident statements that aren’t true.
- Models produce plausible text; when they don’t know, the most plausible text can be invented.
- Rare facts are stored blurrily, but sampling still picks an answer and states it confidently.
- Grading that rewards guesses over "I don’t know" encourages hallucination; trusted sources, tools and checking help.
Check your understanding
Go deeper (more technical)
Researchers distinguish factual hallucinations (wrong about the world) from faithfulness hallucinations (contradicting a document the model was given). RAG helps with the first but can still suffer from the second.
Pre-trained models are often fairly well calibrated: their probabilities roughly match how often they’re right. OpenAI’s GPT-4 report (2023) showed that post-training made that calibration noticeably worse. The idea in the simulation follows Kalai and colleagues (OpenAI, 2025), “Why Language Models Hallucinate”, which argues for grading that penalises confident errors more than abstentions.