The NewsRupt

The case for using a small model on routine work is now strong enough that the interesting question is why so few deployments do it.

Classification, extraction, routing, tagging, short rewrites, structured-output conversion — the unglamorous majority of production AI work — are tasks where a well-chosen small model, given clear instructions and a handful of examples, performs close to a frontier model at a fraction of the cost and a fraction of the latency. On narrow tasks with a fine-tuned small model, it frequently performs better, because the task has been specified precisely and the model has been shown what right looks like.

Latency is the under-discussed half of this. A frontier model answering in two seconds and a small model answering in two hundred milliseconds are not the same product when the call sits inside a user-facing interaction or a loop that runs a thousand times. Many systems that feel slow are slow for this reason and would be improved more by routing than by any prompt engineering.

So why the default to the largest available model. Three reasons, and only one of them is technical.

The first is that prototyping with the strongest model is rational. It removes capability as a variable while the task is still being defined. The error is not starting there; it is never revisiting the choice once the task is stable, because revisiting requires the evaluation harness most teams did not build.

The second is procurement gravity. An organisation with one approved provider, one security review and one billing relationship will use that provider for everything, because adding a second is a quarter of paperwork for a saving that shows up in someone else's budget.

The third is status. Nobody has been criticised for choosing the best model. Choosing a cheaper one and being wrong is a visible decision with a name attached, which is a poor incentive structure for engineering judgement.

The teams that route well tend to do three things. They classify their workload by whether the task requires reasoning across ambiguity or merely reliable transformation, and send only the first category to a large model. They use the large model to generate labelled training data for the small one, which is often the cheapest fine-tuning pipeline available. And they build a fallback path — attempt with the small model, verify cheaply, escalate on failure — which captures most of the saving while bounding the downside.

The limitations here are real and frequently understated by advocates. Small models are more sensitive to prompt phrasing and degrade less gracefully when inputs drift outside the distribution they were tuned on, which shifts maintenance burden onto the deploying team. Fine-tuning creates a versioning and retraining obligation that a prompt against a hosted model does not. Multilingual performance, particularly outside high-resource languages, is markedly weaker in smaller models and this gap is poorly captured by common benchmarks. And a routing layer is itself a system that can fail, adding a failure mode that a single-model architecture does not have.

The claim, then, is not that small models are better. It is that the mapping from task to model is a decision most organisations have never actually made — they made it once during a prototype and have been paying for it ever since. On the volumes that production work now reaches, that unmade decision is usually the largest single line of avoidable cost in the stack, and the one nobody is assigned to review.