How Much Does One AI Response Actually Cost?
Work out what one AI answer really costs to produce, and why idle GPUs and long answers are expensive.
AI companies pay for GPUs by the hour, but sell answers by the token. Getting from one to the other is a division, and it’s surprisingly revealing.
Take one H100 rented in the cloud for roughly $3 an hour. Suppose a well-run engine gets 2,500 tokens per second out of it across all its users. If it were busy every second, that’s 9 million tokens an hour, about 33 cents per million. But real traffic has quiet nights and lunchtime peaks, so the GPU might be properly busy only half the time. The bill doesn’t care: it’s $3 an hour, busy or idle. Now each million tokens costs about 67 cents.
That simple sum shows the three things that drive the price:
- Throughput: everything in this track that squeezes more tokens out of each GPU-hour lowers the cost.
- Utilisation: an idle GPU still costs money, which is why providers chase steady traffic and offer cheaper “batch” APIs for jobs that can wait.
- Answer length: a 500-token answer costs a hundred times more than a 5-token one.
It also explains why API price lists charge separately for input and output tokens, with output several times pricier. Input tokens are read in one parallel prefill pass; output tokens each need their own decode step. Some providers also discount input that repeats a recent prompt, thanks to prefix caching.
The upshot: a typical chat answer costs a fraction of a cent to produce. Small sums per answer, multiplied by billions of answers a day, are why serving efficiency is worth so much engineering.
Key takeaways
- Cost per token ≈ GPU cost per hour ÷ tokens produced per hour.
- Higher throughput and higher utilisation both cut the cost per token; idle GPUs still cost money.
- Output tokens cost more than input tokens, because each needs its own decode step.
- A typical chat answer costs a fraction of a cent, but billions of answers add up.
Check your understanding
Go deeper (more technical)
Real costs include more than GPU rental: electricity, cooling, networking, staff, spare capacity for peaks, and the very large up-front cost of buying the hardware. Providers also blend many workloads on shared fleets to keep utilisation high.
Prices per token have fallen dramatically: for a given level of capability, API prices have dropped by roughly an order of magnitude or more per year in recent years, driven by better hardware, smaller and better models, and the serving tricks in this track.
The simulation’s prices and throughput figures are illustrative.