AI Memory Bandwidth: The Bottleneck Behind Inference Performance

Two accelerators can post nearly identical peak compute numbers and still differ by a factor of several on tokens per second. The spec sheet will not tell you which one wins, because the spec sheet is answering the wrong question. For autoregressive generation, the arithmetic units are already fast enough. The problem is feeding them.

That reframe explains a lot of confusing behavior in decode-heavy, memory-sensitive LLM serving: models that fit comfortably in memory and still crawl, utilization graphs that show an idle chip during a "busy" workload, and vendor bandwidth figures that never survive contact with a profiler. It is not a universal explanation. At low batch sizes, with small models, or in high-concurrency services, queueing, kernel launch overhead, scheduler behavior, and networking can dominate instead. But for the workload that most people actually pay for — generating tokens one at a time — the byte budget is the governing constraint.

This article builds one mental model — bytes moved per token — and uses it to read hardware claims without getting fooled.

The Compute Curve Outran the Memory Curve

The structural reason memory became the bottleneck is a divergence in scaling rates, and it has been running for decades.

Published analysis of the "memory wall" in AI workloads puts the gap in concrete terms: over roughly twenty years, peak server FLOPS scaled at about 3.0× every two years, while DRAM bandwidth scaled at roughly 1.6× and interconnect bandwidth at roughly 1.4× over the same period. The cumulative effect is stark — compute capacity grew by orders of magnitude more than the ability to move data to it.

Treat that as a research signal with a direction, not a precise forecast. The exact multipliers depend on which hardware generations you sample, and the trend line is smoother than any individual product launch. What matters is the mechanism: specialized compute units finish their arithmetic in a handful of cycles, then sit idle waiting for the next operands. Utilization collapses not because the chip is slow, but because it is starved.

Grant the narrow case where compute still dominates. Large-batch prefill, where the whole prompt is processed in parallel, has high arithmetic intensity — many operations per byte fetched — and can genuinely saturate tensor cores. Training with large batches behaves similarly. The memory wall is not a universal law; it is a regime that decode falls into and prefill mostly escapes.

Prefill and Decode Are Two Different Machines

The most common source of bad hardware reasoning is treating "inference" as one workload. It is two, and they pull in opposite directions.

Prefill processes the entire prompt at once. Every token in the input can be attended to in parallel, so the matrix multiplications are large and the arithmetic units stay busy. Prefill is compute-bound, and its cost scales with prompt length.

Decode generates one token at a time, autoregressively. As a first-order model, producing a single token requires fetching the model's weights and the growing KV cache — the stored key and value tensors from all previous tokens — from high-bandwidth memory into the compute cores. Then it computes that one token, and waits to do it all over again for the next one.

The work per token is tiny. The bytes moved are enormous. That is the definition of a memory-bandwidth-bound workload.

The diagnostic handle is arithmetic intensity: bytes moved per FLOP. High intensity means compute is the limiter; low intensity means data movement is. Decode sits at the low-intensity end, which is why a chip with modest peak FLOPS can beat a chip with enormous peak FLOPS on token generation — the faster chip spends more of its time waiting.

Because the two phases want different hardware profiles, a single system cannot be optimized for both without a tradeoff. This is the structural reason disaggregated prefill and decode clusters exist: run each phase on hardware shaped for its bottleneck, and move only the compressed KV cache state between them.

Batching, caching, and concurrency economics are covered elsewhere in this series. The relevant point here is narrower: batching raises arithmetic intensity by amortizing one weight fetch across many sequences, which is why throughput and latency trade against each other. It does not change the underlying byte budget.

Weights, KV Cache, and Activations: What Actually Moves

To estimate a workload's bandwidth demand instead of guessing, you need a byte budget. Three categories dominate.

Weights. In the first-order decode model, the full weight set is read once per token. Their size in bytes — not their parameter count — sets the floor on per-token latency. A model stored in 16-bit precision moves twice the bytes of the same model in 8-bit, and four times the bytes of a 4-bit version. This is the entire mechanism behind quantization speedups on memory-bound workloads: fewer bytes per weight means fewer bytes moved per token. It is not a magic multiplier, and it does nothing for a workload that was already compute-bound.

KV cache. The cache grows with context length and with concurrency, and it competes for the same HBM capacity and the same bandwidth as the weights. Capacity pressure and bandwidth pressure are the same problem wearing two hats: a cache that spills out of HBM does not just consume space, it consumes a much slower link to reach. Longer contexts and more concurrent users both inflate the per-token byte count.

Activations and intermediate tensors. These matter more in prefill and training, where large parallel batches produce substantial intermediate state. In decode, they are comparatively small. Treating all memory traffic as equivalent is a common estimation error — the categories have different scaling behavior and different fixes.

Now make it concrete. Define the estimate as:

bytes_per_token ≈ weight_bytes + kv_cache_bytes_per_token
ceiling_tokens_per_second ≈ achievable_bandwidth / bytes_per_token

The assumptions matter, so state them explicitly. weight_bytes is the total size of the weights in their stored precision — for a 4-bit model, roughly a quarter of the 16-bit figure. kv_cache_bytes_per_token is the per-token cost of the cache, which scales with the number of layers, the number of KV heads, the head dimension, the stored precision, and the number of concurrent sequences sharing the batch. The numerator deliberately omits activation traffic, which is small in decode, and it treats the weight read as a full pass rather than a cache-resident partial read. Both are approximations, and both push the estimate in the same direction: the real ceiling is usually lower than this number suggests.

The comparison that matters is between this ceiling and your measured decode throughput. If measured tokens per second sits well below the ceiling, something other than raw bandwidth is limiting you — scheduling, kernel overhead, or an access pattern that is not achieving peak. If measured throughput is close to the ceiling, you are bandwidth-bound and the only fixes that will help are ones that reduce bytes moved.

The estimate takes minutes, and it immediately reframes a hardware conversation from "how many FLOPS" to "how many bytes per token, and how fast can this device move them."

Why Measured Bandwidth Never Matches the Spec

Peak bandwidth is a theoretical ceiling under ideal access patterns. Achieved bandwidth is what your workload actually gets, and the gap between them is where most performance surprises live.

Access granularity matters. Memory transactions have a minimum size, and requests that do not use the full transaction waste the difference. Bank conflicts, poor locality, and uncoalesced access patterns all reduce effective throughput without showing up anywhere on the datasheet.

Overlap matters. Bandwidth is only useful if transfers proceed while compute runs. A kernel that alternates between fetching and calculating achieves a fraction of the peak that a well-pipelined kernel reaches, even though both move the same total bytes.

Unified-memory and host-offload designs add a second tier. When data spills from device memory to host memory, it crosses a link that is dramatically slower than HBM, and the penalty is not linear — it depends on how often the spill is touched. Offloading extends capacity while adding latency, and that tradeoff should be measured rather than assumed.

Interconnect bandwidth becomes the limiter once a model is sharded across accelerators. Tensor parallelism splits weights across devices, which relieves per-device memory pressure but replaces it with communication pressure: every layer boundary now requires a collective operation such as an all-reduce, in which each device contributes its partial result and every device receives the combined one. That collective sits on the critical path. Past a certain shard count, the communication cost dominates and adding devices makes things slower, not faster.

Thermal and power limits can throttle memory clocks. The same chip behaves differently in a dense rack than on an open bench, which is one reason benchmark numbers from controlled conditions do not always transfer to production.

The practical signature to look for: profile achieved bandwidth and accelerator utilization together. Low utilization combined with high memory throughput is the fingerprint of a bandwidth-bound workload. If you see that pattern, adding compute will not help. Reducing bytes moved will.

Reading Hardware Claims Without Getting Fooled

With the model in hand, vendor claims become auditable. The discipline is separating three things that marketing prose tends to blend: confirmed facts (published specs, reproducible measurements), vendor claims (marketing bandwidth, "X times faster"), and your own inference about your workload.

Ask what workload the claim was measured on. Batch size, sequence length, precision, and concurrency change the answer more than the chip does. A tokens-per-second figure without those parameters is not a performance claim; it is a component spec wearing a benchmark's clothing.

Watch for claims that quietly switch between memory capacity and memory bandwidth. They solve different problems. Capacity determines whether a model fits and how much KV cache you can hold. Bandwidth determines how fast you can stream what fits. A device can be generous on one and stingy on the other, and the marketing copy rarely makes the distinction explicit.

Watch for precision laundering. A bandwidth number quoted without a precision is incomplete, because the effective bytes moved per token depend on the weight format. A 4-bit model moves roughly a quarter of the bytes of its 16-bit equivalent, so the same device can appear four times faster on one workload than another — not because the hardware changed, but because the byte budget did. Recent accelerator designs that support low-precision formats natively attack the bottleneck directly by shrinking the tensors that must be fetched, and they avoid paying extra compute or memory traffic to convert values back up. That is a real mechanism, not a marketing trick — but it only helps workloads that were bandwidth-bound to begin with.

The decision rule: estimate bytes moved per token for your workload, divide by achievable bandwidth, and compare that ceiling to the vendor's number before believing the benchmark. If the vendor's figure exceeds your computed ceiling, either your estimate is wrong or the claim is measured on a different workload. Both are worth knowing.

What to Measure First

The model is only useful if you run it against your own workload. Start with one real model you serve and estimate its bytes per token: weights plus KV cache, divided by achievable bandwidth. It takes minutes and permanently changes how you read hardware announcements.

Then measure. Profile achieved bandwidth and accelerator utilization on the same workload to confirm which resource is actually saturated. Build a small comparison harness that varies batch size and context length so you can watch the prefill/decode boundary move — the point at which a workload stops being compute-bound and starts being bandwidth-bound is the most useful thing you can know about your own system.

The skills worth developing next follow directly from the model: profiling tools that expose memory throughput rather than just utilization, quantization tradeoffs and their accuracy costs, KV cache management under long contexts and high concurrency, and interconnect-aware sharding decisions.

Where the Bottleneck Moves Next

Relieving memory bandwidth pressure does not remove the constraint. It relocates it, and the new location is worth predicting in advance.

Lower precision reduces bytes moved per token, but it pushes the system toward a compute limit or an accuracy limit — at some point the arithmetic units become the bottleneck again, and quantization stops buying speed. Sparsity reduces effective work, but it depends on hardware support and on whether the model's structure actually exhibits exploitable sparsity. Speculative decoding exploits idle compute during decode by drafting multiple tokens and verifying them in parallel, but its benefit is bounded by the acceptance rate of the draft model; a poor draft wastes the compute it was meant to use.

Offloading and tiered memory extend capacity while adding latency. The tradeoff is explicit and should be measured, not assumed — the question is whether the added latency is hidden by overlap or exposed on the critical path.

One market signal is worth watching without over-reading it. Memory manufacturers are committing substantial research and manufacturing investment toward high-bandwidth memory, and AI infrastructure buildout is absorbing a growing share of memory supply. That is context, not evidence for the workload model above, and the timeline and magnitude remain genuinely open. If memory availability tightens, system cost and capacity planning feel it before token throughput does.

The open questions worth tracking: how much of the gap closes through architecture changes — better caching, smarter data placement, new memory hierarchies — versus memory technology itself, and whether workload shapes shift toward compute-bound regimes as models and serving patterns evolve. Frame those as scenarios, not predictions.

The durable rule is not a spec to memorize. It is a procedure: optimize the resource that is actually saturated, and re-measure after every change, because the bottleneck migrates the moment you relieve it. Estimate bytes per token. Divide by achievable bandwidth. Treat that number as the ceiling every hardware claim has to beat — and then go profile your own workload, because the only benchmark that matters is the one running on your hardware, with your model, at your context length.