Capability was the easy part
I’ve been looking hard at one pattern as AI usage explodes inside large organisations: we need to get a grip on AI spending.
I coach U11 girls’ handball. You learn the same thing three matches: one player is better than everyone else, and every instinct in your body says keep her on the court. She wins the ball back. She scores. Then it’s the second half of the third match in two days, she has nothing left in her legs, and the situation you actually needed her for is the one she can’t do anything about.
Knowing who your best player is takes no skill at all. Everyone in the hall can see it, (the parents loudest of all). The skill is knowing which five minutes need her—and having the discipline to hold her back until they arrive.
Enterprise AI is stuck on the same problem, and most companies are still playing their star every minute of every match. For three years the question was which model is smartest. That was the easy part—and it’s mostly settled. The question that now decides who makes money from AI and who just spends on it is a much cheaper one: which task needs which model, at what cost, under whose control. Here’s what I’ve learned watching that shift up close.
The habit we built
Nobody trusted the small models and the bills were rounding errors, so reaching for the most capable model every single time was the right call. It removed the risk of being wrong and cost almost nothing to over-buy. Both of those conditions are gone. The habit stayed.
Written down it looks absurd. You would not send a cardiac surgeon to take a blood-pressure reading. He would get it right, obviously. He is also slower, scarcer and vastly more expensive, and the nurse’s number is identical. Companies do the surgeon thing with AI all day long, spending frontier prices on work a small model finishes for a fraction of the cost and the energy, and they file it under prudence.
Good enough is now a number
Why is this a 2026 problem and not a 2028 one? Because two curves crossed: “good enough” became measurable, and prices fell faster than quality rose.
Stanford’s Scaling Intelligence Lab and Together AI ran twenty-plus models across eight hardware accelerators and published it as intelligence per watt. Models at 20B parameters or under, running locally, answer 88.7% of single-turn chat and reasoning queries correctly. Nine in ten. On hardware that fits under a desk. Efficiency improved 5.3x from 2023 to 2025 and most of that came from software, not silicon, which is the part I’d underline if I were buying GPUs this year. Nvidia’s own researchers landed in the same place from the other direction: small models should be the default for agent work, roughly 10–30x cheaper to run, with a strong model held in reserve for the genuinely hard calls. When the company selling the GPUs tells you to buy smaller, check the incentive. Then notice they’re still right.
Prices fell faster than quality rose. Stanford’s AI Index put GPT-3.5-level output at $20 per million tokens in late 2022 and $0.07 two years later. 280-fold. A paper from this March tracking inference pricing 2020 to early 2026 puts the whole decline near 600x and gives it a shape: economy tier halves every 1.10 years, mid tier every 1.55. Faster than Moore’s law. And the decomposition says it comes from software and architecture, with hardware contributing roughly nothing, which should worry anyone whose AI strategy is a GPU purchase order.
The bill already flipped
Here is the number that moved this up my own priority list, and it landed last week. Gartner has enterprise inference spending overtaking training spending for the first time: $23.3 billion against $19 billion in 2026, inside an AI-optimised infrastructure market of $42 billion that nearly doubled in a year. By 2027 inference is 59% of it.
AI stopped being a research line and became an operating line. Research lines get approved once and forgiven when they overrun. Operating lines get metered, then cut. Nobody sent a memo about the switch. The invoice sent it.
Routing is the skill
If most work runs fine on a cheap or local model and a minority genuinely needs the frontier, then the money sits in one decision: which is which. That decision is routing.
Not theory. Berkeley’s RouteLLM cut cost by over 85% against sending everything to GPT-4 while holding 95% of the quality on MT-Bench, because the strong model was actually required on 14% of calls. One query in seven.
Routing is a business decision disguised as a model call.
This ships as product now. OpenRouter’s Pareto router takes one quality score in the request and sends it to the cheapest model that clears the bar, re-ranking the shortlist as new models land. Gartner tracks the same function as the AI gateway and expects the share of teams building multi-model applications who run one to go from a quarter in 2025 to seven in ten by 2028.
The market has noticed, loudly. Fortune reported this month that OpenRouter is in acquisition talks with Stripe at around ten billion dollars, while Salesforce and Databricks build routing into their platforms and Cursor, Ramp, Meta and Runway build their own. Same piece has the reason: 62% of organisations hit by unexpected AI costs big enough to change a business decision, 40% escalated to the board, a third imposing emergency spending freezes.
So routing goes from clever trick to owned infrastructure, fast. Two hundred applications each making their own model choice gives you no control and no leverage. Make the choice once, at the platform, and you have both. If governance matters, it’s also the cheap place to enforce data residency and privacy rules, because “which model” and “which jurisdiction” are the same question asked twice.
Then meter the tokens
Cheaper per task is not cheaper in total. This is where people get caught. A company I know of burned its annual model budget in four months. It did not have a model problem. It had a metering problem.
It’s becoming a discipline with a name and a headcount. The FinOps Foundation, the people who taught companies to control cloud spend, surveyed 1,192 practitioners in February. 98% now manage AI spend, up from under a third two years ago, and 78% report into the CTO or CIO. Their first rule is the one above: stop sending everything to the most expensive model, match the model to the task.
The metric I’d push for is cost per unit of work. Think “return on tokens” (ROT), per use case. Budget compute like you budget anything else, make consumption visible to the people doing the consuming, review what it returned. NB: measure output against tokens and not tokens alone, otherwise you’ve built an incentive to burn compute to look busy. And ask the uncomfortable version: which team could halve its token spend next quarter without anyone outside noticing? If nobody can answer, you don’t have a model problem either.
Unmetered AI is a governance gap, not a technology one. Gartner’s forecast that more than 40% of agentic AI projects get cancelled by end of 2027 names cost and weak controls as the cause. Not capability.
The part nobody puts on the slide
Two things spoil the tidy version of this story, and I’d rather say them than have a CFO find them.
First. If cheap models handle the easy 70% for almost nothing, then that 70% is worth almost nothing too. Value moves to the hard minority that still needs the frontier, and to the people who can tell which is which. The March pricing paper puts a number on the ceiling the floor can’t reach: reasoning-tier models price at an average of 31.5 times standard, and flagship pricing simply refuses to decay the way the economy tier does. Efficiency at the base and a widening premium at the top are one trend seen from two ends.
Second, older than AI, Jevons’ paradox. Make a resource cheaper and people use far more of it. Bain’s numbers lay it out: token costs halved between December 2024 and December 2025 while token consumption grew 450%. Halve the price, multiply the volume by five and a half, and the invoice nearly triples. Apollo’s chief economist calls it Jevons in action and points out that spending on large language models has doubled since late 2025 while the unit cost of intelligence collapsed.
Promise your board that efficiency shrinks the AI bill and demand will quietly make a liar of you. The honest promise is a better cost per unit of work, plus a defensible reason behind every expensive call you make.
The teams that come out of the next two years ahead will treat model choice as a discipline rather than a reflex, and they’ll be able to answer one plain question about every frontier call in the stack: why this model, and not a cheaper one?
Three questions I’d expect every AI leader to answer in 2026:
What share of calls route to small/local vs. frontier—and why?
What’s our ROT trend by use case, and what did we stop because it didn’t clear the bar?
Where do governance and routing meet to enforce privacy and residency by design?
What would you add? Where has routing or metering saved you real money without denting outcomes—and where did it bite? I’d love to learn from your examples. #AI #FinOps #DigitalTransformation #Leadership
Sources
Saad-Falcon, Narayan et al., Intelligence Per Watt: Measuring Intelligence Efficiency of Local AI, Stanford Scaling Intelligence Lab and Together AI, November 2025. arXiv:2511.07885 · project page
Du, M., Tiered Super-Moore’s Law: Price Evolution, Production Frontiers, and Market Competition in Large Language Model Inference Services, March 2026. arXiv:2603.28576
Stanford HAI, AI Index Report 2025 (inference cost decline) and AI Index Report 2026
Gartner, Worldwide AI-Optimized IaaS Spending to Grow 96% in 2026, 10 August 2026
Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027
Ong et al., RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing, LMSYS / UC Berkeley
OpenRouter, Pareto Router documentation
Fortune, Why every company wants an AI model router right now, 9 August 2026
Fortune, Tokens are getting cheaper, but AI costs keep climbing anyway, 17 June 2026 (Torsten Slok; Bain consumption figures)
FinOps Foundation, State of FinOps 2026 · Linux Foundation announcement, February 2026
Belcak et al., Small Language Models are the Future of Agentic AI, NVIDIA Research, 2025