Understanding Tokens, Limits and API Costs
Learn how tokens drive an app’s costs and limits, and the main ways to keep both under control.
Everything an app does with a model is measured in tokens, roughly ¾ of an English word each (Track 2). They set the bill, and they set the limits. To finish this track, let’s do the sums an app builder actually has to do.
The bill. APIs charge per million tokens, with separate prices for input (everything you send) and output (everything the model writes), output typically costing several times more (Track 4 explained why). A single request costs a fraction of a cent. The surprises come from how requests add up.
The biggest surprise is chat. The model is stateless (Track 2), so every turn re-sends the whole conversation: system prompt, every earlier message, plus the new one. Turn 20 pays again for turns 1 to 19. Input per turn keeps growing, and the total for a long chat grows much faster than its length. Multiply by thousands of users and long conversations can dominate the bill.
The limits. Three kinds catch out new apps. The context window: input plus output must fit, so very long chats eventually have to drop or summarise old turns. The maximum output you set: hit it and the reply is cut off mid-sentence. And rate limits: requests and tokens per minute per account, beyond which the API answers 429 until you slow down.
The main levers for keeping costs down:
- Summarise old turns instead of re-sending them in full.
- Prompt caching: providers charge much less for the unchanged start of a prompt they processed recently, thanks to the prefix caching from Track 4.
- Shorter replies where they’re enough, with a sensible length cap.
- A smaller model for the easy majority of requests.
That completes the toolkit for building on a model. Next, Track 6 tackles a model’s biggest blind spot: how to give it knowledge it was never trained on.
Key takeaways
- APIs bill input and output tokens separately; output costs more per token.
- Chats re-send the whole conversation every turn, so long chats get expensive fast.
- Limits: the context window, the maximum output length, and rate limits (429 errors).
- Levers: summarise history, use prompt caching, cap replies, and use smaller models where possible.
Check your understanding
Go deeper (more technical)
Token counts depend on the model’s tokenizer, so the same text can cost different amounts on different models. Providers publish tokenizer tools to count in advance; the response’s usage field gives the exact numbers afterwards, and apps log them to watch costs.
For a chat where every turn adds t tokens, the total input over n turns grows roughly with n²·t/2: double the conversation length and its input cost roughly quadruples. Prompt caching discounts vary by provider, often to around a tenth of the normal input price for cached tokens. Batch APIs, for jobs that can wait hours, are often about half price.
The simulation’s prices, sizes and limits are illustrative.