How AI Breaks Language Into Tokens
See how text becomes the numbers a model can work with, and why words get cut into pieces.
Here’s an awkward fact: a neural network can’t read. It only does arithmetic on numbers. So before your message reaches the model, a tokenizer chops it into tokens and swaps each one for an ID number from the model’s vocabulary.
Why pieces? Look at the alternatives. One token per word sounds natural, but English alone has hundreds of thousands of words, before you add names, typos, slang, code and a hundred other languages. Anything missing from the list would be unreadable. One token per letter fixes that, but makes every text about four times longer and each token nearly meaningless on its own.
Sub-word tokens split the difference. The most common recipe, byte-pair encoding, learns its pieces from data: start with single characters, find the pair that sits side by side most often (say “t” + “h”), glue it into a new token, and repeat tens of thousands of times. Frequent chunks like “the”, “ing” or “ capital” end up as single tokens; “strawberry” may still need three.
This humble step has big consequences. Prices, speed and length limits are all counted in tokens, roughly ¾ of an English word each. And because the model sees “str·aw·berry” rather than ten letters, a question like “how many r’s are in strawberry?” is surprisingly hard for it.
Key takeaways
- Models read tokens, not letters or words. Each token is an ID number.
- Common words become one token; rare words are split into several pieces, so any text can be written.
- In English, one token is about 4 characters, or roughly ¾ of a word.
- Cost, speed and length limits are all counted in tokens, and tokenization explains some odd weaknesses.
Check your understanding
Go deeper (more technical)
Modern tokenizers usually run BPE on bytes (UTF-8) rather than letters, so any text, emoji or language can be encoded without “unknown” tokens. Vocabulary sizes: GPT-2 used 50,257 tokens; Llama 3 uses about 128,000.
Languages that are underrepresented in the tokenizer’s training data need more tokens for the same meaning, which makes them slower and more expensive to process. Numbers are another quirk: a year like “2024” may be one token while a long number is chopped into irregular chunks, one reason arithmetic on long numbers is tricky.