Why the Same Prompt Can Produce Different Answers
Learn how the model turns probabilities into one actual choice, and what temperature and top-p really do.
Every pass ends with a probability for every token in the vocabulary. Something still has to turn that list into one actual choice. That step is called sampling, and a couple of dials control it.
The obvious strategy is to always take the top token, known as greedy decoding. It’s ideal when there’s one right answer, like pulling a date out of an invoice. For open-ended writing it’s dull and oddly fragile: always playing safe produces bland text and can get stuck repeating itself.
So most chatbots roll weighted dice instead: a token with a 30% probability gets picked about 30% of the time. That’s why asking the same question twice can give two different answers, and why one may be better than the other.
temperature reshapes the dice before the roll. Low temperature sharpens the distribution, so the favourite becomes even more of a favourite; at 0 you’re back to greedy. High temperature flattens it and gives long shots a real chance: more surprising, more creative, and more likely to wander into nonsense. Temperature doesn’t make the model smarter or dumber. It only changes how adventurously it picks from what it already believes.
top-p is the safety net. It keeps only the most likely tokens that together reach p, say 90%, and throws away the long tail of weird options before rolling. With it, you can turn the temperature up for variety without letting “sand” into your cup of tea.
Key takeaways
- The model produces probabilities; sampling settings decide how one token is chosen from them.
- Temperature 0 = always the top choice. Higher = more variety and more risk.
- Top-p cuts off the unlikely tail, so higher temperatures stay sensible.
- Random sampling is why the same question can get different answers each time.
Check your understanding
Go deeper (more technical)
The raw scores are called logits. Probabilities = softmax(logits / T). As T → 0 this becomes “argmax” (greedy decoding); T = 1 uses the model’s own distribution; T > 1 flattens it.
Other common knobs: top-k (keep only the k most likely tokens), min-p (drop tokens much less likely than the best one), and repetition or frequency penalties. Even at temperature 0, results can vary slightly between runs, because GPU arithmetic in different batch sizes and orders isn’t perfectly identical.