The economics of inference: why cost per token became a board-level number

The line item that arrives after launch

Teams model AI projects like software projects: a build cost, then a maintenance cost. AI systems do not behave that way. Every request has a marginal cost that does not fall as you scale — it rises, linearly, with usage. That makes inference a cost of goods sold, and it belongs in the unit economics from the first design conversation, not in a surprised finance review two quarters after launch.

The pattern is predictable. A pilot with a few hundred users costs almost nothing, and nobody thinks about it. Adoption succeeds. Volume grows twentyfold, the workflow gets chattier, someone adds a reasoning step, and the monthly bill becomes a topic at the executive meeting. The system is working exactly as intended, which is precisely the problem.

The margin inversion

Traditional software has near-zero marginal cost, so the heaviest user is the most profitable one. AI features on a per-seat price invert that: your most engaged customer consumes the most inference and can quietly become your least profitable account. Several categories of AI product have run into this, and the responses are all imperfect — usage tiers, credit systems, rate limits, or aggressive cost engineering.

The strategic implication is that pricing and architecture are the same conversation. If your commercial model cannot absorb variable cost per interaction, your architecture has to drive that cost toward zero, and that constraint should shape the design from the start.

The levers, roughly in order of return

The good news is that inference cost is highly compressible. In practice these are the moves that pay:

• Model routing. Most production traffic is easy. Send it to a small, cheap model and reserve the large one for genuinely hard requests. A classifier deciding which is which is often the single highest-return component in the system.
• Caching. Semantic caching for repeated questions, and prompt caching for the long static preamble you resend on every call. Enterprise traffic is far more repetitive than teams assume.
• Smaller and specialized models. A fine-tuned small model frequently matches a much larger general one on a narrow task at a fraction of the cost and latency.
• Quantization. Running in lower precision cuts memory and cost substantially, usually with negligible quality change on well-defined tasks.
• Context discipline. Retrieving five well-chosen chunks instead of fifty is cheaper, faster, and typically more accurate. Bloated context is expensive twice.
• Batching and scheduling. If you self-host, keeping the accelerator busy is most of the economics.

Applied together on a system that was never optimized, halving cost is a routine outcome and larger reductions are common — usually with quality holding or improving, because the discipline that reduces cost also reduces noise.

Why falling prices do not save you

Per-token prices have fallen dramatically and will probably continue to. It is a mistake to plan on that solving the problem, for two reasons.

First, demand expands to meet the lower price. Cheaper inference makes new use cases viable, so consumption grows faster than unit cost falls and the total bill increases. Second, the frontier moves toward approaches that spend more compute per request — longer reasoning, multi-step agents, larger contexts — which pushes in exactly the opposite direction. The floor drops; what you put on the floor gets heavier.

Build, rent, or both

At low and spiky volume, serverless per-token pricing is almost always correct: no capacity to manage, no idle cost, immediate elasticity. As volume becomes steady and predictable, dedicated capacity generally wins on unit cost, and at large sustained scale self-hosting open-weight models can be dramatically cheaper — provided you have the engineering capability to keep utilization high, because idle accelerators are the fastest way to make self-hosting more expensive than the API you left.

The pragmatic architecture is a portfolio: serverless for burst and experimentation, dedicated capacity for the steady core, and the option to self-host where volume and compliance justify it. What makes that portfolio possible is a gateway layer and portable weights — so moving a workload is a routing change rather than a rebuild.

What to do about it now

Three habits separate teams that control this from teams that get surprised. Instrument cost per outcome from day one, broken down by workflow and by customer, so the conversation is specific. Set an explicit budget per interaction during design, the way you would set a latency budget. And run a quarterly optimization pass — the levers above are not one-time fixes, and models, prices, and traffic patterns all move.

Inference economics is not a finance problem that arrives later. It is an architecture problem you are solving now, whether or not you are looking at it.