What is inference, and why does it cost what it costs?
Inference is running a trained model to produce output. Unlike training, which is a one-off capital cost, inference is a recurring cost paid per request. Providers price it per million input and output tokens, and output tokens usually cost several times more than input tokens because each one requires a separate forward pass through the model. Reasoning models, which generate large numbers of hidden intermediate tokens, can cost far more per answer than their headline price suggests.
A request is tokenised, loaded into the model's context, and processed in two phases. Prefill processes the whole prompt in parallel and is compute-bound. Decode then produces output one token at a time and is memory-bandwidth-bound, which is why output is the expensive half. Providers batch many requests together to keep accelerators busy, and cache repeated prompt prefixes so the prefill work is not repeated — cached input is typically billed at a tenth of the normal rate or less.
A support assistant with a 4,000-token system prompt answering 100,000 questions a month pays for 400 million input tokens before a single answer is written. Moving that prompt into a cached prefix is usually the single largest cost reduction available.
Inference economics decide whether an AI feature has a viable unit cost. Two teams using the same model can differ by an order of magnitude in cost per resolved ticket purely through prompt design, caching, batching and model routing.
- That the advertised per-token price predicts your bill. Reasoning tokens, retries and long system prompts dominate real spend.
- That a cheaper model always lowers total cost. A weaker model that needs two attempts and a human check can cost more.
- Provider pricing changes frequently; any figure should be read with its collection date.
- Latency and throughput vary by region, batch size and time of day, so published benchmarks rarely reproduce exactly.