What is an inference provider?
An inference provider runs models on its own accelerators and sells access per token or per GPU-hour. The category spans three different businesses: serverless token APIs for open-weight models, dedicated endpoints with reserved capacity, and raw GPU rental. They compete on price per token, tokens per second, time to first token, model coverage and the contractual questions — data retention, region, uptime — that decide whether an enterprise can use them at all.
Providers batch incoming requests, keep model weights resident in accelerator memory, and use serving stacks that manage the key-value cache efficiently. Throughput and latency depend on hardware, batch policy and current load, which is why the same model served by two providers can differ severalfold in speed and cost. Quantisation — serving a compressed version of the weights — lowers cost and can change output quality, and is not always disclosed prominently.
A team serving an open-weight model moves from a general cloud GPU rental to a serverless token API, cutting idle capacity cost but accepting variable latency at peak times.
For anyone running open-weight models, provider choice is now a first-order cost and latency decision, and a compliance one. It is also reversible in a way model choice often is not, provided the application is written against a portable API surface.
- That identical model names mean identical outputs. Quantisation and serving settings differ between providers.
- That published tokens-per-second figures hold under load. Most are measured on single requests.
- Prices and available models change monthly; any comparison needs a collection date.
- Independent latency measurements vary by client region and time of day.