AI Inference Costs: When Model Routing Becomes the Real Product Decision

The prototype proved the idea. Now the bill decides whether the idea gets to exist.

Every AI product starts the same way. You pick the strongest model available, put everything the model might need into the prompt, and run one call to confirm the concept works. That instinct is correct for a prototype. The problem is what happens next: those defaults quietly become the production architecture, and a pattern designed to answer "can this work?" is now responsible for answering "can this scale economically?"

That is the trap worth opening up. Not because per-token pricing is fake, but because it is the wrong unit for the decision most teams are actually making. The number that determines viability is cost per successfully completed outcome. Routing is one lever that moves it — but only after simpler reductions are exhausted and the workload is diverse enough to justify the added system.

The Prototype Default That Becomes Your Cost Structure

A prototype is optimized for possibility. Production is optimized for repeatability under a budget. The same defaults serve opposite goals.

Consider what one "outcome" actually costs. In a simple chat feature, one user request maps to one model call, and per-token price is a reasonable planning metric. In an agent loop — where the model plans, calls tools, observes results, and decides what to do next — one user request becomes many model calls. Every turn, every retry, every wrong tool call is billed. Waste multiplies across the loop, and the loop is where the cost lives.

Cost per request and cost per completed outcome are different numbers. Cost per request tells you what one call cost. Cost per completed outcome tells you what the business actually paid across every turn and retry it took to get a usable answer. Only the second one maps to whether the feature can survive contact with real users at real volume.

The narrow case is real: for a low-volume internal tool with a handful of users and a narrow task, per-token price is genuinely sufficient. You do not need a routing layer to summarize twelve documents a day. The moment volume, task diversity, or agent loops enter the picture, the metric stops being adequate.

What Actually Drives Inference Cost

Inference is the process of running data through a trained model to produce an output. It is the stage that generates revenue, as opposed to training, which generates capability. When people talk about AI inference costs, they usually mean the money spent serving requests in production.

The cost stack underneath a single request is wider than the model's price list:

• Hardware and energy. Accelerators cost money to buy and power to run. Energy efficiency — performance per watt — is a real line item at scale.
• Utilization and batching. A GPU serving one request at a time is a GPU mostly idle. Batching multiple requests together raises throughput per unit of hardware.
• Quantization and KV cache. Reducing numeric precision and caching attention state both cut the compute and memory needed per token.
• The software stack. Serving frameworks, schedulers, and memory management determine how much of the hardware's capacity you actually capture.

Three metrics connect that stack to spend. Throughput is how many tokens the system produces per unit of time, often measured in tokens per second. Latency splits into time to first token (how long before the response starts) and time per output token (how fast it streams after that). Goodput is the throughput achieved while still meeting your target latency levels — the number that matters when you care about both speed and volume.

For the routing decision specifically, the relevant distinction is this: throughput, batching, quantization, and the serving stack are variables you optimize inside a single deployment. Latency and cost per model are the inputs a router uses to choose between deployments. Goodput is the metric that tells you whether a routing policy is actually holding its latency promise under load.

There is a useful framing here that a 2025 research paper on inference economics calls the "impossible trinity": model quality, inference performance, and economic cost cannot all be maximized at once. Any deployment is a balance point among the three, not a maximum on all three axes. That is not a limitation to complain about. It is the shape of the design space.

Directional context is encouraging but should not be overread. NVIDIA, citing Stanford's 2025 AI Index, reports that inference cost for a system performing at GPT-3.5 level dropped more than 280-fold between November 2022 and October 2024, with hardware costs declining roughly 30% annually and energy efficiency improving about 40% per year. Those are vendor-relayed figures about a specific capability level, not a promise about your workload. Your prompts, your tool schemas, and your traffic mix are not the benchmark conditions.

Why Price Per Million Tokens Misleads You

The list price is a starting point, not a decision. Four things break the naive comparison.

Verbosity differs across models. A cheaper model that writes more tokens, or makes more tool calls, can cost more per task than a pricier model that answers in fewer tokens. NVIDIA's routing documentation makes this explicit: each model has its own verbosity profile, and that profile covers tool calls, not just output tokens.

Retries and failures erase price advantages. A model that needs two attempts to get something right is not cheaper than a model that gets it right once. VentureBeat's coverage of a Gemini Flash release makes the point directly: a model that costs less per token but requires substantially more retries may not ultimately be cheaper.

Introductory pricing expires. Promotional rates are vendor claims with an expiry date. That same Gemini Flash release cut API prices in half through the end of 2026 — a real discount, and a real planning hazard if you build a margin on it. Plan against standard rates.

Benchmarks are results under agreed test conditions. Your repository, your prompts, your tool schemas, and your failure modes are not those conditions. A leaderboard score tells you how a model performed on someone else's task set.

The decision rule that follows is simple and unglamorous: benchmark candidate models on your own task set and compare cost per successful task, not list price. If you have not built that task set, you do not yet have the evidence to choose.

When Routing Beats Picking One Model

Model routing means dispatching each request to a model chosen for that request, behind a single endpoint, instead of sending everything to one model. The router sits between your application and a pool of models, and it makes a per-request decision.

Routing addresses two failure modes at once. Send everything to a frontier model and routine requests pay frontier-model economics. Send everything to a small model and complex requests lose quality. A router tries to hold both ends: cheap models for the routine, capable models for the hard.

What signals can a router use? NVIDIA's routing guidance groups them into three areas:

• Model capabilities — which models can solve the task correctly.
• Model cost profiles — the latency and cost associated with each model.
• Infrastructure signals — load, errors, and other system-level state that enables reliable handoffs.

On top of those signals sits a policy layer. The router scores candidate models by blending predicted accuracy with cost, latency, or other deployment constraints, then sends the request to the best tradeoff for that workload. Some routers classify the request first — by topic or estimated difficulty — and match it to a model. Others route only difficult problems to expensive models and let everything else run cheap.

Routing is becoming a visible product pattern among infrastructure and enterprise vendors. Microsoft's Azure documentation describes a model router that assesses each incoming request and dispatches it in real time, with routing modes that prioritize cost, quality, or a balance of the two, plus built-in failover. Ramp, a corporate expense platform, launched its own routing service in 2026 with strategies for routing based on benchmarks or difficulty. These are market signals about productization, not evidence of broad adoption or proven economics. The adoption test that matters is still your own workload.

Where routing does not pay: low request volume, a narrow task distribution where one model handles everything well, or any situation where the routing overhead and added failure modes exceed the savings. A router is a system you now have to operate. If your traffic is small and uniform, that system is pure cost.

The Tradeoffs Routing Cannot Remove

Routing is an engineering commitment with its own failure surface. Four costs are structural.

Quality risk. A misrouted hard request degrades output silently. A crash is loud; a slightly worse answer is not. This is the failure mode that scares me most, because it is hard to detect without per-request evaluation.

Latency risk. Classification and handoff add time before the first token. Failover adds more. If your product's value depends on responsiveness, the router's overhead is part of your latency budget.

Operational surface. Multiple providers, multiple pricing models, fallback paths, and version drift across models. Every model in the pool is a dependency that can change under you.

Evaluation burden. You now need per-request visibility into which model served the request and whether quality held. Without that, you cannot tell whether routing helped or quietly hurt.

There is a fifth cost that is not technical. Routing services may retain inputs, outputs, and tool calls by default. Ramp's router, for example, records model inputs, outputs, and tool calls for one year by default, with personally identifiable information removed before use. That is a compliance decision, not a cost decision — and it belongs in the same conversation as the price.

The Levers Around Routing

Routing is one lever among several, and the cheapest wins usually come before the router. Each of the following is either a pre-routing intervention, a substitute for routing, or an input the router needs.

Caching (pre-routing). Prompt caching reuses computation for repeated prompt prefixes. Semantic caching goes further, matching requests by meaning rather than exact text. Both reduce work before any routing decision is made.

Prompt and agent optimization (pre-routing). Trimming context, tightening tool descriptions, and removing unnecessary turns cut cost at the source. An agent that takes a wrong turn burns tokens on turns that should never have happened.

Deployment shape (substitute). Provisioned throughput trades commitment for predictable capacity. Batch processing trades latency for cost. A fine-tuned small model can beat a general model on a narrow task — and if it handles your whole distribution, it replaces the router entirely.

Observability (input). You cannot tune what you cannot see. The per-request signals that make routing safe are input and output tokens, cache hit rate, latency, the model that actually served the request, and evaluation scores. Two numbers matter most: cost per request and cost per completed outcome.

The sequencing judgment is the one I would defend hardest. Delete and simplify the workflow before automating the routing of it. A router that dispatches efficiently between models is still dispatching a wasteful loop.

A Staged Adoption Test

The criteria above collapse into one sequence. Run it in order, and stop at the first stage that clears your quality bar at acceptable cost.

Stage 0: Measure the baseline. Instrument cost per completed outcome across the full loop, including retries and tool calls. Without this number, every later stage is guesswork.

Stage 1: Remove redundant work. Cache repeated prefixes, trim context, tighten tool descriptions, and delete unnecessary agent turns. Re-measure. Many teams find the routing question disappears here.

Stage 2: Test a single-model baseline. Benchmark two or three candidate models on your own task set. If one model clears the quality bar at acceptable cost across your whole distribution, you are done. Do not build a router to solve a problem you do not have.

Stage 3: Add routing only if the workload is heterogeneous. The signal to proceed is a measured gap: no single model clears the bar economically across your task mix, and the mix has distinguishable difficulty tiers. Then route, and require the routing layer to beat the Stage 2 baseline on cost per completed outcome after accounting for routing overhead.

Stage 4: Re-evaluate on every model release cycle. Model releases arrive on short cycles — one Gemini Flash release landed three weeks after its predecessor. A deployment that was optimal last quarter may not be now.

If Stage 3 does not produce a measured gain, revert. A router you cannot justify with a number is a router you should not operate.

What to Measure Before You Commit

If you reach Stage 3, here is the evaluation plan I would run before committing to a routing architecture.

Build a task set from real traffic. Include the messy and ambiguous cases, not curated demos. The demos are the cases your prototype already handles.

Define the quality bar first. Then measure cost and latency only on configurations that clear it. A cheaper configuration that fails the bar is not a cheaper configuration.

Track cost per completed outcome across the full loop. Include retries and tool calls. This is the number that maps to your unit economics.

Instrument the serving model per request. Routing decisions must be auditable after the fact. If you cannot answer "which model served this request, and did quality hold?", you cannot improve the router.

Set a re-evaluation trigger. Tie it to model releases and to any shift in your traffic mix, not to a calendar quarter.

What This Means for Your Product Roadmap

Routing is a product decision because it sets three things your pricing must survive: the quality floor your users experience, the latency budget your interface can promise, and the unit economics your margins depend on. Those are not infrastructure details. They are the terms of the product.

The skills worth building now are cost instrumentation, evaluation harnesses, and request classification. None of them are exotic. All of them compound — a task set you build this quarter is the same task set that tells you whether next quarter's model release is worth adopting.

I want to be honest about where the evidence is thin. Published cost comparisons come largely from vendor benchmarks and single-workload studies. The 280-fold decline figure describes a specific capability level under specific conditions. Treat these as signals about direction, not as transferable numbers for your workload. The open questions are real: how durable promotional pricing turns out to be, whether routing services commoditize into a thin layer, and how much of the theoretical savings survives real production traffic with real failure modes.

Here is the closing question I keep returning to. If your margin depends on a model you do not control, what part of the system are you actually building? The router, the evaluation harness, the task set, the observability layer — those are yours. The model is rented. Build the part that stays.

The decision rule is not complicated. Pick the cheapest configuration that clears your measured quality bar. Instrument cost per completed outcome. Re-test on every model release cycle. Routing is not a one-time model swap; it is a durable capability — an evaluation and observability loop that keeps your economics honest as the models underneath you change.