AI Inference Capacity Planning: Traffic, Queues, and Latency Budgets

A capacity plan built from average QPS and a single latency target will survive the spreadsheet and die on the first traffic spike. The number was never the problem. The assumption behind it was.

Most teams size inference capacity the way they size a web tier: measure average requests per second, multiply by a per-request cost, add a safety margin, ship. That works when service time is roughly constant and arrivals are smooth. Neither holds for LLM inference. Service time swings with output length, prompt length, and cache state. Arrivals cluster. And the thing users actually feel — queue wait — grows non-linearly as utilization climbs. The plan that looks comfortable at 60% utilization can collapse at 85%.

This is not a sizing exercise. It is a falsification exercise. Every number in a defensible capacity plan should come with the measurement that would prove it wrong. I want to walk through how to build that plan: what to extract from a workload trace, how to turn a latency target into per-stage budgets, how to convert concurrency into GPU count, and which observations should force you to tear the plan up and start again.

Why Average Traffic Is the Wrong Input

Capacity planning for inference is a queueing problem. The inputs are the arrival distribution and the service-time distribution. Mean throughput is an output, not an input.

Two services can serve the same 10 million requests per day and need wildly different capacity. One receives them at a steady 115 per second. The other receives 80% of them in a three-hour window, with bursts that briefly hit 20× the daily mean. The daily total is identical. The queue depth is not. The second service needs enough capacity to absorb the burst, or enough queue tolerance to let requests wait, or a load-shedding policy that drops the excess. Those are three different architectures with three different cost profiles.

Service time is the other half of the problem, and it is not constant either. For a transformer model, the time to serve one request depends on how many input tokens need prefill, how many output tokens get generated, and whether the prefix was already cached. A request that hits the KV cache skips most of the prefill work. A request that misses it pays full price. Same endpoint, same user, different cost depending on history.

The bridge between traffic and capacity is Little's Law: average concurrency equals average arrival rate multiplied by average time in system. If 100 requests arrive per second and each spends an average of 2 seconds in the system, you need roughly 200 concurrent slots on average. That equation is why latency and capacity are the same conversation. You cannot set a latency target without implicitly setting a concurrency requirement, and you cannot size concurrency without knowing how long requests actually take.

There is a narrow case where averages work: strictly rate-limited batch workloads with no interactive latency SLO. If you are running overnight document classification and the only requirement is that the batch finishes by morning, then mean throughput is a fine input. The moment a human is waiting for a token, the mean stops being sufficient.

Reading a Workload Trace Before You Size Anything

Before you pick a GPU count, you need to know the shape of your traffic. If you have production data, you have a trace. If you are pre-launch, you have a projection, and you should treat it as a hypothesis rather than a fact.

The first split that matters is input tokens versus output tokens. Prefill and decode have different cost curves. Prefill processes the entire prompt in parallel and scales with input length. Decode generates tokens one at a time and scales with output length. A workload with long prompts and short outputs is prefill-heavy. A workload with short prompts and long outputs is decode-heavy. They stress different parts of the serving stack and they respond differently to batching. If you collapse them into a single "tokens per request" number, you lose the ability to predict which resource will run out first.

From there, pull percentiles, not means. You want p50, p95, and p99 of prompt length, output length, and the gap between consecutive request arrivals. The p99 output length is a major driver of tail decode demand, because the longest generations consume the most decode time and the most KV cache memory. The p99 inter-arrival gap tells you how bursty your traffic really is. A service with a p99 gap of 50 milliseconds and a p50 gap of 2 seconds has a burst problem that the daily average will never reveal.

Cache hit rate belongs in the trace analysis as a first-class input, not an afterthought. If a meaningful share of your requests share a common prefix — a system prompt, a document, a conversation history — those tokens can be served from the KV cache instead of recomputed. A higher cache hit rate skips prefill for those tokens, which lowers time-to-first-token and reduces the GPU capacity you need for the same traffic. NVIDIA's sizing guidance treats cache hit rate as a primary input for exactly this reason. If your plan assumes a 60% hit rate and production delivers 30%, you have over-provisioned throughput you will never see.

Diurnal and weekly patterns matter more than the daily total. Most consumer-facing AI services have a morning ramp, a midday plateau, an evening peak, and a weekend trough. The burst multiplier — peak QPS divided by mean QPS — is the number that drives your headroom requirement. A service with a 2× multiplier needs very different capacity than one with a 10× multiplier, even at the same daily volume.

What a trace cannot tell you is just as important. It cannot tell you about traffic you have not launched yet. It cannot tell you about the retry amplification that appears only under load, when timeouts trigger client retries that double your effective request count. And it cannot tell you how users will change their behavior once the feature is fast enough to use casually. Treat the trace as a lower bound on the shape of your future traffic, not a ceiling.

Building the Latency Budget Backwards

A latency target is not a number infrastructure picks. It is a contract with the product. "Sub-second first token" and "sub-second full response" are different promises with different capacity implications. The first constrains prefill. The second constrains prefill plus decode plus queue wait plus network.

Decompose end-to-end latency into four stages: queue wait, prefill (which produces time-to-first-token, or TTFT), decode (inter-token latency, the gap between successive output tokens), and network plus overhead. Each stage consumes part of the budget, and each stage is constrained by a different resource.

TTFT and inter-token latency have different user-perceived costs. A chat interface tolerates a slower first token less than slower streaming. Users notice the pause before the response starts. Once tokens are flowing, a slightly slower stream is often invisible. A code completion tool is the opposite: the first token needs to be fast, and the stream needs to be fast too, because the user is reading as it arrives. A batch summarization job does not care about either, only about total completion time. The budget allocation should follow the interaction pattern, not a generic "low latency" goal.

Percentile targets are what force headroom into the plan. A p50 latency target is easy to hit and tells you almost nothing. A p99 target is what determines how much spare capacity you carry, because the p99 request is the one that arrives during a burst, with a long prompt, on a cold cache, while the GPU is already busy. If your plan is sized to meet p50, it will fail p99 by a wide margin. The p99 budget is the one that costs money, and it is the one worth arguing about explicitly.

Three things routinely break the budget in ways the plan did not allocate for. Long-context requests consume prefill time proportional to input length, and a single very long request can consume disproportionate KV-cache capacity relative to a short one. A cold cache — after a deploy, a cache eviction, or a new user with no history — pays full prefill cost on every request until the cache warms. Cross-region routing adds network latency that is invisible in a single-region benchmark but very visible to a user on the other side of an ocean. Each of these consumes budget you did not set aside, and each should have an explicit line in the plan.

The latency budget is a product decision because it determines cost. Tightening a tail latency target generally requires more headroom, and headroom is idle capacity you pay for. The exact amount depends on your workload and serving stack, and it must be measured through load testing rather than assumed from a rule of thumb. That tradeoff belongs in a conversation between product and infrastructure, not in a number infrastructure chooses alone.

From Concurrency to GPU Count

Concurrency is not a setting you pick. It is a consequence of arrival rate and service time. You can cap it — most serving frameworks let you set a maximum batch size or a maximum number of in-flight requests — but the cap is a policy choice, not a plan. The plan is the number of GPUs required to serve the offered load within the latency budget.

The first thing to internalize is that utilization is not a target to maximize. Queue wait grows non-linearly as utilization approaches saturation. At 50% utilization, a request might wait a fraction of its service time. At 80%, the wait grows noticeably. At 95%, small fluctuations in arrival rate produce large fluctuations in queue depth. The exact curve depends on the arrival and service-time distributions, but the shape is universal: the last 10% of utilization is where latency goes to die. A plan that targets 90% utilization is a plan that will miss its p99 target the first time traffic is burstier than expected.

Batching raises throughput but adds queue wait. If you wait to accumulate a batch of 8 requests before running them, the first request in the batch waits for the other 7 to arrive. That wait is real latency, and it comes out of the budget. The tradeoff between batch size and queue wait is a policy decision, and it should be made explicitly rather than inherited from a framework default. Larger batches improve GPU efficiency and lower cost per token. Smaller batches improve latency. You cannot have both, and the right point on the curve depends on whether your users are waiting interactively or running a batch job.

The real concurrency ceiling on a given node is usually memory, not compute. Each in-flight request holds a KV cache — the stored attention state for its context — and that cache consumes GPU memory proportional to context length. A node with plenty of compute headroom can still refuse new requests because it has run out of memory to hold their KV caches. This is why concurrency planning and memory planning are the same exercise. If you size for compute and ignore memory, you will hit a wall that your throughput math never predicted.

The practical framing is core-and-flex. A core capacity — GPUs you own or reserve on a long contract — covers your baseline load. A flex capacity — on-demand or spot instances — absorbs bursts. The split follows traffic predictability. Stable, predictable traffic justifies long-term commitments because the per-hour cost is lower. Volatile or experimental workloads benefit from flexible capacity because you are not paying for idle reserved GPUs during the trough. The contract length decision is downstream of how well you can forecast your traffic, not upstream of it.

A worked example makes the arithmetic concrete — and shows where the arithmetic stops. Suppose your trace shows a peak arrival rate of 50 requests per second and a measured mean service time of 400 milliseconds. Little's Law gives a mean concurrency of 50 × 0.4 = 20 requests in flight. That is the average. It is not the number of slots you need, because the p99 request does not arrive at the average rate with the average service time. It arrives during a burst, with a longer prompt, on a colder cache, while the node is already busy.

To find the real slot count, replay the trace against a single node and measure two things: the maximum concurrency the node can hold in memory before it starts rejecting or queuing, and the p99 latency the node delivers at each offered load level. If the node saturates at 40 concurrent requests and the replayed trace pushes offered concurrency above that during bursts, you need more nodes — but the number comes from the replay, not from multiplying a p99 latency budget by a peak arrival rate. The headroom factor between mean concurrency (20) and provisioned concurrency (whatever the replay demands) is the cost of meeting p99 rather than p50. It is not waste. It is the price of the latency contract, and it is a number you measure rather than assume.

Cost per Request Is a Capacity Output, Not an Input

The number that matters for unit economics is cost per successful request, not cost per GPU-hour. Cost per GPU-hour is an input to that number. It is not the number itself.

Cost per request equals provisioned capacity cost divided by successful requests served. That division is where idle headroom becomes a real cost line. If you provision for peak and run at 40% average utilization, you are paying for 60% idle capacity, and that idle capacity shows up in the cost per request whether you account for it or not. This is the honest version of the utilization story: high utilization lowers cost per request, but it also raises queue wait, which raises latency, which may raise retries and fallbacks, which raises the effective request count. The relationship is not monotonic. There is a utilization range where cost per request is minimized, and it is not 100%.

Retries, timeouts, and fallbacks multiply the effective request count. A request that times out and retries counts twice against your capacity but once against your success metric. A request that falls back to a smaller model after a failure consumes capacity on two models. If your plan assumes one request per user action and production delivers 1.3, your cost per success is 30% higher than the plan predicted, and your capacity headroom is 30% smaller. This is the retry amplification that a trace from staging will never show you, because staging does not fail under load the way production does.

Reserved versus on-demand versus spot is a contract-length decision that follows traffic predictability. Reserved capacity is cheaper per hour but you pay for it whether you use it or not. On-demand is more expensive per hour but you only pay when you use it. Spot is cheapest but can be reclaimed, which makes it suitable for batch workloads with checkpointing and unsuitable for interactive services with tight latency budgets. The right mix depends on how confident you are in your baseline forecast. If your baseline is stable and well-measured, reserve it. If it is a guess, rent it until the guess becomes a measurement.

Power and facility limits can become a hard capacity ceiling in dense deployments. A rack can only dissipate so much heat, and a data center has a finite power budget. Microsoft Research's bottom-up analysis of inference energy use estimates a median of 0.34 Wh per query for frontier-scale models under realistic workloads, rising roughly 13-fold for test-time scaling scenarios with 15× more tokens per query. Those figures are estimates under specific assumptions, not universal constants, but the planning implication holds: power draw per GPU translates into how many GPUs you can actually run in a given facility. A capacity plan that ignores power is a plan that will hit a wall the spreadsheet never modeled.

The margin question is the one that decides whether the feature is viable. At what utilization does cost per successful request exceed the revenue per successful request? That is the break-even point, and it is a function of your pricing, your quality bar, and your traffic shape. If the break-even utilization is 70% and your traffic peaks at 50% average, the feature does not have a margin problem — it has a capacity planning problem, and the fix is either to raise utilization by consolidating workloads or to lower the provisioned capacity by accepting a looser latency target.

The Measurements That Would Invalidate Your Plan

A capacity plan is a set of assumptions wearing a spreadsheet. The plan is only as good as your willingness to let the measurements break it. Here are the specific observations that should force a revision, and how to instrument for them.

Cache hit rate below assumption is the single most common way a plan overstates achievable throughput. If you assumed 60% and you are getting 30%, your prefill load is roughly double what you planned for, and your TTFT will be worse than predicted. Instrument cache hit rate per endpoint, per model, and per user segment. A drop after a deploy or a prompt change is a capacity event, not just a performance regression. If cache hit rate drops, revise your prefill demand assumption and re-derive TTFT.

Output length distribution drifting longer than the trace predicted is the second most common failure. Users adapt to faster models by asking for more. A summarization feature that started with 200-token outputs may drift to 500-token outputs as users learn what the model can do. That drift increases decode time, increases KV cache memory pressure, and increases cost per request. Track the p50 and p99 output length over time, not just at launch. If output length drifts, revise your decode time and KV-cache memory assumptions.

Queue wait growing while utilization looks moderate indicates a bottleneck elsewhere. If your GPU utilization is 60% but queue wait is climbing, the constraint is probably memory (KV cache exhaustion), network (interconnect saturation), or the scheduler (batching policy). Utilization is a symptom, not a diagnosis. When queue wait and utilization disagree, trust the queue wait and go looking for the real bottleneck. If queue wait rises at moderate utilization, inspect memory, scheduler, and network before adding GPUs.

Retry and fallback rates under load that never appear in staging are the third common failure. Staging does not reproduce production traffic patterns, production failure modes, or production client behavior. Instrument retry counts, timeout rates, and fallback invocations as first-class capacity metrics. A retry rate of 5% is a 5% capacity tax that your plan did not budget for. If retry rate grows, revise your offered load and cost-per-request assumptions upward.

Silent degradation on low-traffic endpoints is the failure that monitoring tuned for peak traffic misses. A low-traffic endpoint can degrade for hours without triggering a peak-oriented alert, because its absolute numbers never cross the threshold. The arXiv study on production inference incidents found that low-traffic endpoints and rollout windows were recurring sources of operational failures, precisely because standard monitoring does not watch them closely. Use dynamic thresholds or anomaly detection on per-endpoint latency and error rates, not just global aggregates. If a low-traffic endpoint degrades, revise your monitoring thresholds and add it to the capacity review.

The right posture is to treat capacity as an SLO with automated saturation signals rather than a one-time sizing exercise. The same arXiv study recommends treating capacity as a first-class SLO: forecast token and throughput growth, tie capacity policies to saturation signals surfaced in request lifecycle trends, and automate capacity increase decisions to avoid manual hotfixes that inflate time-to-mitigation. A capacity plan that is not continuously re-derived from observed data is a plan that is quietly going stale from the day it was written.

What to Learn Next

The loop that makes this work is instrument, measure, re-derive, repeat. If you are starting from zero, the order matters.

Instrument first. Before you build any sizing model, get token-level metrics, queue depth, and cache hit rate flowing from a single service. You cannot plan capacity for a system you cannot see. The metrics that matter most are the ones this article keeps returning to: p50 and p99 of prompt length, output length, and inter-arrival gap; cache hit rate; queue wait; and retry rate. Those six numbers are a minimum starting set for this method, not a complete production observability contract. If you have them, you can build a defensible plan. If you do not, you are guessing.

Build a small load generator that replays a real trace shape rather than synthetic constant load. Constant load tells you nothing about queueing behavior, because queueing behavior is a function of variance. Replay the burst pattern, the output-length distribution, and the cache-hit pattern from your production trace. Then watch what breaks. The failure you find in a load test is cheaper than the failure you find in production.

Practice the arithmetic on one service before generalizing to a fleet. The worked example in this article — arrival rate, service time, measured saturation point, headroom factor — is the whole method compressed into four numbers. Run it on your busiest endpoint. Then run it on your least busy endpoint and see whether the same headroom factor applies. It usually does not, and the difference is where the interesting decisions live.

Where this connects to adjacent decisions: routing, cascades, and model selection all change the inputs to this plan. A routing policy that sends easy requests to a small model and hard requests to a large one changes the service-time distribution. A cascade that falls back on failure changes the retry amplification. A model swap changes the prefill and decode cost curves. Each of those is a capacity planning decision wearing a different hat.

The boundary where this approach stops working is highly variable agentic workloads with unbounded token counts. An agent that decides how many tool calls to make, how many documents to read, and how long to reason before answering does not have a stable service-time distribution. The trace from last week does not predict the trace from next week, because the workload itself is changing. For those systems, capacity planning shifts from sizing to admission control: you set a budget per task, you cap the tokens per task, and you shed or queue tasks that exceed the budget. The queueing math still applies, but the service-time distribution is a policy you enforce rather than a property you measure.

Start with one service. Instrument cache hit rate, queue wait, and output-length distribution. Re-derive the plan from what you observe. Then decide which measurement would break it — and go looking for that measurement before it finds you.