LLM Serving: How Batching, Caching, and Concurrency Change Unit Economics

Your dashboard is green. GPU utilization sits in a healthy band, average latency looks fine, and nobody has filed a complaint this week. Then the invoice arrives, and cost per request has climbed again. The p99 line has started to drift upward in a way that is easy to explain away and hard to reverse.

That combination — healthy averages, rising unit cost, creeping tail latency — is rarely a hardware problem. It is a memory allocation problem wearing a performance costume.

This article assumes you already understand why inference is memory-bound and why utilization matters more than peak specs. We are going one layer up: from silicon to scheduler. The claim I want to defend is that batching, caching, and concurrency are not three independent knobs. They are three ways of spending the same currency — KV-cache memory — and every serving decision is an allocation choice with a price in throughput, tail latency, or output quality.

The Memory Budget Behind Every Serving Decision

Start with the physical constraint, because every tradeoff downstream inherits from it.

Model weights occupy a fixed floor. A 7-billion-parameter model loaded in 16-bit precision consumes roughly 14 GB before a single request arrives. That number does not move with traffic. It is rent you pay whether the endpoint is busy or idle.

The KV cache is different. It is allocated per request and grows per token. The size per token scales with the number of layers, the number of attention heads, the head dimension, and the precision in bytes. Multiply that by context length, then by concurrent requests, and you get the real memory pressure of a live endpoint.

The practical consequence is blunt: batch size is not a free parameter. It is bounded by how much memory remains after weights and activations are accounted for. When you increase batch size, you are not "using the GPU harder." You are spending KV-cache headroom that some other request will need later.

So frame the endpoint as a fixed pool divided three ways: weights, KV cache, and headroom. Headroom is not waste. It is the buffer that absorbs bursty arrivals, long contexts, and the occasional request that generates far more tokens than your median. Every optimization in this article moves memory between those three buckets. None of them creates memory from nothing.

Why Static Batching Wastes the GPU You Already Paid For

The naive approach is static batching: collect a group of requests, run them together, wait for the whole group to finish, then start the next group.

The failure mode is not subtle. Language models generate variable numbers of tokens per request. A chatbot reply might be 40 tokens; a document summary might be 900. In a static batch, every request waits for the longest one. The short requests finished their useful work hundreds of decode iterations ago, but their slots stay occupied until the slowest sequence emits its final token.

That is not primarily a latency problem. It is a utilization problem. Idle decode slots are GPU-seconds you paid for and did not use. A batch that is 30% idle is a batch you are paying full price for and running at two-thirds efficiency.

Continuous batching — also called in-flight batching — attacks this directly. Instead of waiting for the whole batch, the runtime evicts finished sequences at each decode iteration and admits new requests immediately. The batch becomes a rolling window rather than a fixed cohort.

Vendor-reported gains here are large. Anyscale has demonstrated up to 23x throughput improvement using continuous batching versus static batching, measured on OPT-13B on an A100 40GB across varying concurrency levels. Microsoft's enterprise guidance describes continuous batching pushing GPU utilization from 30–40% to 80%+ on comparable node pools.

Treat those numbers as directional evidence, not constants. They are workload-specific claims measured under particular conditions. Your traffic has a different output-length distribution, a different prompt-length distribution, and a different arrival pattern. Re-measure on your own load before you budget against someone else's multiplier.

There is also a failure mode worth naming explicitly: continuous batching improves throughput, but under bursty load without admission control it can worsen per-request latency. The scheduler keeps admitting work because it can, and the queue grows faster than it drains. Throughput looks excellent right up until your p99 crosses the threshold your product promised.

Caching: Prefix Reuse, KV Reuse, and What Each One Actually Saves

"LLM response caching" gets used to describe at least two different mechanisms that save different things. Collapsing them leads to bad decisions.

Prefix caching reuses the KV state for a shared prompt prefix. If every request in your system begins with the same 1,200-token system prompt, few-shot block, or retrieved-context header, that prefix does not need to be recomputed from scratch on every call. The engine computes it once, keeps the KV state, and reuses it.

What does prefix caching save? Prefill compute. It reduces time-to-first-token (TTFT) and the GPU work spent processing the input. It does not reduce decode cost — the per-token generation work is unchanged. If your bottleneck is long outputs rather than long inputs, prefix caching will not rescue you.

Semantic or response caching is a different animal. It skips inference entirely for repeated or near-repeated queries. It saves the whole request, which is why it looks so attractive on a cost chart. It also introduces a correctness risk that does not appear in any latency dashboard: a near-match is not a match. If your similarity threshold is loose, you will serve a stale or subtly wrong answer, and the user will have no way to know the system decided their question was close enough.

That is a quality regression disguised as a performance win. It will not show up in TTFT, throughput, or utilization. It shows up in user trust, weeks later, when someone notices the assistant keeps giving an answer that was correct for a different question.

There is a memory-management dimension too. KV-cache reuse across requests requires block-based or paged allocation to avoid fragmentation. Without it, the cache hit rate gains get eaten by allocation overhead and wasted blocks. Paged attention exists precisely because naive contiguous allocation does not survive real concurrency.

The decision boundary is clean. Prefix caching pays off when prompts share long, stable prefixes. It is close to worthless when every request has a unique prefix — you pay the bookkeeping cost and get nothing back. Semantic caching pays off when query volume is high and repetition is genuine, and it is dangerous when queries are superficially similar but semantically distinct.

One open question worth flagging: cache invalidation policy for semantic caches is still largely hand-rolled. There is no standard correctness metric for it, and no widely adopted practice for deciding when a cached answer has gone stale. If you deploy one, you are inventing your own policy.

Concurrency Limits Are an Economic Control, Not a Safety Valve

Most teams treat concurrency limits as protection — the thing that stops the server from falling over. That framing undersells them. Concurrency is the primary lever that sets your throughput-latency frontier, and it is coupled to batch size through KV-cache memory.

More concurrent requests means less memory available per request. That caps the context length you can serve, or forces preemption: the scheduler evicts a request's KV state to make room, then recomputes it later when the request resumes. Preemption and recomputation are the hidden cost of pushing concurrency too high. Throughput looks fine while tail latency and wasted compute both rise, because you are paying to compute the same tokens twice.

The degradation curves for TTFT and time-between-tokens are not linear, and they do not degrade at the same rate. Research on serving frameworks shows the curve can collapse past a threshold rather than bend gracefully. ScaleLLM's concurrency benchmarks, for example, show TTFT and time-between-tokens rising sharply as concurrent request counts increase, with different slopes for different serving stacks. The shape of that curve is the thing you need to measure, not the peak number.

Mixed workloads make this harder. Latency-sensitive chat traffic and latency-tolerant batch jobs — report generation, offline summarization, embedding backfills — compete for the same memory pool. Microsoft's SageServe work on Office 365 traffic, which characterizes one of the first public Internet-scale LLM serving workloads at over 10 million requests per day, describes routing and placement decisions across multiple timescales rather than a single global concurrency setting. Short-term request routing and long-term GPU scaling are separate control problems, and co-optimizing them produced up to 25% savings in GPU-hours against their baseline, with an 80% reduction in GPU-hour wastage from inefficient autoscaling.

That is the shape of the problem: concurrency is not one number. It is a policy that depends on what mix of traffic you are serving and what latency each class actually needs.

My practical rule: set concurrency from your p99 latency target and your memory budget, then verify with a load test that includes realistic output-length variance. Do not set it from a peak-throughput benchmark. Peak throughput is measured at the point where latency has already degraded past what your users will tolerate.

Where the Serving Stack Hides Your Cost

If you attribute all serving cost to the model forward pass, you will optimize the wrong layer.

A production endpoint has at least three layers. Infrastructure orchestration manages GPU nodes, networking, and container lifecycle. A serving or orchestration layer handles request routing, autoscaling, replica placement, and streaming. The inference engine executes forward passes and manages KV cache. Each layer can be the bottleneck, and each has a different fix.

Gateway and routing overhead is real and measurable. End-to-end latency decomposes into gateway time plus engine time, and the gateway share grows with request volume and streaming behavior. ScaleLLM's end-to-end breakdown exists specifically because the gateway is not free — it is a component with its own latency, concurrency handling, and resource efficiency characteristics.

Autoscaling is where the money usually leaks. LLM load does not express well in CPU or memory metrics. It is expressed in tokens and requests. If your autoscaler is watching the wrong signal, you get both failure modes at once: over-provisioning during quiet periods and cold-start latency spikes when traffic arrives. Microsoft's production study reports GPU-hour savings from co-optimizing routing and placement rather than tuning the engine alone — which is a strong hint that for many deployments, the scheduling layer dominates the engine layer in cost impact.

Here is my interpretation, stated as interpretation: for most teams below a certain request volume, fixing autoscaling and replica placement beats micro-optimizing kernels. Kernel-level work has a ceiling and requires deep specialization. Scheduling work has a much higher ceiling and can be validated with a load test. If you have limited engineering hours, spend them where the leverage is.

Measuring the Trade Without Lying to Yourself

Every optimization in this article can be validated or faked. The difference is measurement discipline.

Track these as a set, not individually: cost per successful request, TTFT, time-between-tokens, p50/p95/p99 latency, cache hit rate, preemption and recompute rate, and GPU utilization. Any one of them in isolation will mislead you. High utilization with rising p99 is a regression. Low cost per request with a falling cache hit rate is a configuration that will not hold.

Average latency is the most common lie in serving dashboards. A configuration that improves mean throughput while doubling p99 is a regression for any interactive product. Users do not experience your mean. They experience the request that took four seconds while they were deciding whether to close the tab.

Quality has to be measured alongside cost, and this is the part teams skip. Caching, quantization, and aggressive batching can all change outputs. Without an eval set, the regression is invisible until it reaches users. OpenAI's model optimization guidance makes the same point from the platform side: LLM output is non-deterministic, model behavior changes between snapshots, and developers have to measure continuously rather than assume stability.

Change one variable at a time and hold the workload fixed. Mixed changes make attribution impossible and produce the classic outcome where the team "optimized" the system and it got slower, with no way to identify which change did it.

The anti-pattern to name directly: tuning against a synthetic benchmark with uniform output lengths, then deploying against real traffic with heavy-tailed generation lengths. Uniform benchmarks reward configurations that fall apart the moment one request generates 2,000 tokens. Your load test needs the tail, or it is measuring a system you do not operate.

Choosing an Optimization Order That Compounds

Given all of the above, here is the sequence I would follow, with the boundaries where each step stops paying.

First, fix utilization and admission control. If your GPUs sit below roughly half utilization, the cheapest win is scheduling, not a smaller model. Continuous batching and a real admission policy cost engineering time, not quality. This is the highest-leverage step and the one most teams under-invest in.

Second, add prefix caching where prompt structure supports it. It reduces prefill cost without changing the output distribution, which makes it one of the few optimizations with no quality risk. Audit your prompts first: if they do not share long stable prefixes, skip this step rather than forcing it.

Third, tune concurrency against a stated p99 target. Not a peak-throughput target. Write down the latency number your product promises, then find the concurrency level that holds it under realistic load.

Fourth, and only then, consider model-size or quantization changes. These carry the largest quality risk and the largest potential savings, which is exactly why they belong last. You want a working eval set before you touch them, not after.

The overkill boundary matters too. For low-volume internal tools with generous latency budgets, continuous batching and multi-tier caching add operational complexity that outweighs the savings. If your endpoint serves a few thousand requests a day and nobody minds waiting, the correct configuration is the simple one. Complexity has a maintenance cost that does not appear on the GPU invoice.

And treat any configuration as a snapshot. Engine defaults and hardware generations change quickly. What is optimal today is a measurement, not a permanent setting.

What to Learn Next and What to Watch

The skills that pay here are unglamorous and specific. Learn to load-test an endpoint with realistic length distributions rather than uniform ones. Learn to read engine metrics — queue depth, preemption rate, cache hit rate — and connect each one to a mechanism. Build a small eval set that runs on every serving change, so quality regressions surface before users find them.

Tooling in this space is maturing. Google Research's PROMPTS work is an early signal that automated bottleneck diagnosis is becoming practical: it describes a multi-agent framework that synthesizes profiler data and proposes optimized configurations, reporting improvements across eight production workloads and matching the configuration human engineers ultimately adopted in most cases. Treat that as a research result, not a shipped product guarantee. But the direction is clear — configuration search is becoming a tool rather than a specialty.

Two signals worth watching. First, whether accelerated serving modes and specialized hardware partnerships shift the cost curve enough to change the batching-versus-latency tradeoff itself. Second, whether cache correctness and invalidation practices standardize, because right now every team deploying a semantic cache is inventing its own policy and hoping.

The decision rule that survives all of this: pick the serving configuration that meets your stated p99 target at the lowest cost per successful request, and re-measure whenever traffic shape or model version changes. Serving configuration is an economic choice bounded by KV-cache memory. The right answer is not the fastest benchmark or the cheapest invoice — it is the one you can defend with a load test and an eval set.

Before you change any serving parameter, build those two things. They are the difference between optimizing and guessing.