Local AI Inference: Hardware, Privacy, and Performance Tradeoffs

The model loads. The first prompt returns in two seconds. Then the context grows, a second request arrives, and the whole thing crawls.

That symptom is the fastest way to understand what actually governs local AI inference. It is not raw compute. It is memory capacity first, memory bandwidth second, and operational ownership third. Everything else — quantization choices, runtime selection, hardware class — is downstream of those three constraints.

This article is for developers deciding whether to run models on their own hardware or private infrastructure. It assumes you already understand what a transformer is, what inference means, and roughly how a serving stack is assembled. The goal is narrower: to help you predict which constraint will bind before you buy hardware or write the deployment.

The Constraint You Actually Hit First

LLM inference has two phases, and they stress different parts of the machine.

Prefill processes the input prompt. It is compute-heavy and parallelizable: the model can attend to all prompt tokens at once, so the work spreads across available compute units. This is why time to first token (TTFT) often looks acceptable even on modest hardware.

Decode generates output tokens one at a time. Each token depends on the previous one, so the sequence is inherently serial. Every decode step requires reading the model weights and the KV cache from memory. The compute is trivial relative to the data movement. Decode is memory-bandwidth-bound.

This distinction explains the most common misdiagnosis in local inference. A developer sees slow token generation, assumes the GPU is too weak, and buys more FLOPS. The actual bottleneck is bandwidth — how fast the accelerator can move weights and cache into the compute units. More compute does not fix a bandwidth wall.

Capacity comes before speed. A model must fit in memory before any performance question matters. Three things compete for the same pool:

• Model weights. At FP16, a 7B-parameter model needs roughly 14 GB just for weights. At 4-bit quantization, that drops to roughly 3.5–4 GB.
• KV cache. Every token in the context window requires stored key and value tensors. The cache grows linearly with context length and with the number of concurrent requests.
• Runtime overhead. CUDA context, framework buffers, and fragmentation consume memory that does not appear in model-size calculations.

The KV cache is where most capacity planning goes wrong. A model that fits comfortably at 4K context may fail at 32K. A model that serves one user fine may run out of memory with five concurrent requests, because each request carries its own KV cache. Single-request benchmarks systematically overstate real serving capacity.

Unified memory versus discrete VRAM. Apple silicon and devices like NVIDIA's DGX Spark use unified memory, where CPU and GPU share a single pool. This raises the capacity ceiling — a 128 GB unified-memory device can hold models that would not fit on a 24 GB discrete GPU — but unified memory commonly offers lower bandwidth than discrete high-bandwidth memory. Discrete GPUs have a hard VRAM ceiling but move data faster within it. The failure modes differ too: unified memory often degrades more gradually as it fills, while discrete VRAM tends to hit an out-of-memory error. These are common patterns, not rules — measured bandwidth, usable memory, paging behavior, and runtime support decide the result on a particular device.

The observable signal tells you which constraint you are in. Fast first token, slow subsequent tokens: decode-phase bandwidth starvation. Slow first token: prefill compute or a very long prompt. Out-of-memory at longer context: KV cache capacity. Model loads but refuses to start: weights do not fit.

Quantization: The Lever With a Price Tag

Quantization reduces the numeric precision of model weights — and sometimes activations — to shrink memory footprint and reduce bytes moved per token. It is the primary lever for fitting a model into available memory.

The format families matter because they map to different runtimes:

• GGUF formats (Q4_K_M and similar) target llama.cpp-style runtimes. NVIDIA's developer guidance recommends Q4_K_M checkpoints for llama.cpp deployments.
• NVFP4-class formats target vLLM and PyTorch paths on NVIDIA hardware.
• INT4 and INT8 ONNX target edge and NPU deployments. Microsoft's Foundry Local documentation describes INT4 quantization reducing model size to roughly 25% of the original while maintaining about 95% of performance — a vendor-reported figure.

That 95% claim deserves scrutiny. It is an aggregate across benchmarks, not a guarantee for your task. Quantization degrades unevenly: long-context reasoning, structured output, code generation, and multilingual tasks tend to suffer more than short-form chat. Aggregate benchmarks hide this because they weight task categories in ways that may not match your workload.

The practical rule: quantize to fit, then measure on a task-specific evaluation set before trusting the fit. If Q4 degrades your specific task below acceptable accuracy, the model does not fit — regardless of what the aggregate number says.

Throughput, Latency, and the Batch Size Trap

Latency and throughput are different objectives that pull hardware in different directions.

Latency is what a single user feels: time to first token and inter-token latency. Throughput is tokens per second across all requests. A configuration optimized for one may perform poorly on the other.

Continuous batching and paged attention raise throughput by sharing the accelerator across concurrent requests. Instead of processing requests one at a time, the serving engine interleaves them, keeping the accelerator busy. This is how production serving stacks achieve high utilization.

But each concurrent request consumes KV cache. Throughput gains trade directly against context capacity. A laptop that feels instant for one user becomes unusable for five because the same memory pool now holds five KV caches at the target context length.

Speculative decoding — using a small draft model to propose tokens that a larger model verifies — improves decode speed without changing the fundamental bandwidth ceiling. Kernel-level optimizations do the same. NVIDIA reports that llama.cpp optimizations delivered up to 1.9x higher throughput on a GeForce RTX 5090, attributed to kernel improvements, speculative decoding, and faster prefill. These are vendor-reported figures and useful as directional signals, not as guarantees for your hardware and workload.

What to measure before committing:

• Tokens per second at your target concurrency, not at batch size 1.
• Time to first token at your target context length.
• Memory headroom under sustained load, not at idle.

Privacy Is a Boundary, Not a Feature

The privacy argument for local inference is straightforward: prompts, retrieved context, and outputs never cross a network boundary you do not control. That is a real architectural property, not a marketing checkbox.

But the boundary leaks in predictable places:

• Telemetry and crash reporting. Inference runtimes and agent frameworks may phone home by default.
• Cloud fallback routing. A system configured to escalate hard queries to a hosted model has a network boundary — it is just conditional.
• Model download and update channels. Weights and runtime updates arrive over the network.
• Agent tool calls. An agent that queries external APIs, fetches URLs, or writes to cloud services has outbound traffic regardless of where inference runs.

If you cannot enumerate every outbound network call in your stack, you do not have a local privacy boundary. You have a local inference step inside a distributed system.

Hybrid routing is the honest middle ground. Route sensitive or high-volume queries locally and escalate hard queries to a frontier model. Research on local-first routing — including a Stanford study measuring "intelligence per watt" across local models and accelerators — reports substantial energy and cost reductions under idealized query-to-model assignment. Those numbers assume perfect routing decisions and single-query inference. Treat them as an upper bound on what routing could achieve, not a forecast for your deployment.

Compliance adds its own complications. Local execution simplifies some data-handling questions and complicates others: device management, key custody, audit trails, and model provenance all become your responsibility.

The Operational Bill Nobody Quotes

Hardware amortization versus per-token API pricing is the visible comparison. The invisible one is larger.

Local inference can win on cost at high sustained volume, but that is a hypothesis to test, not a general outcome. It loses on bursty or low-volume workloads, where the hardware sits idle but the capital cost does not. The marginal cost of a local token approaches zero — but only if the hardware is already owned, powered, cooled, and idle enough to absorb the load, and only if the local model's quality is comparable for your task.

The hidden line items:

• Model updates and re-quantization when new checkpoints arrive.
• Runtime and driver version churn.
• Evaluation harness maintenance to verify that updates did not regress quality.
• On-call for a machine in an office.
• Engineering hours to assemble and maintain a stack that hosted APIs provide as a service.

The ecosystem is reducing assembly friction. Packaged local stacks — Perplexity's Portable Computer, announced in September 2026, bundles model, inference engine, agent harness, tools, and a security sandbox into a single application — shift effort from assembly to operation. NVIDIA's PAIR tool, also announced in September 2026, pools idle machines on a local network, routing inference requests to whichever system has capacity. These reduce the integration burden. They do not eliminate the operational one, and they are early signals of packaging momentum rather than proof that local inference has become broadly economical.

Build a cost model before committing: sustained tokens per month, acceptable latency, privacy tier, and the fully loaded hourly cost of the engineer who maintains it. If the local option only wins when that engineer's time is valued at zero, it does not win.

Choosing Hardware Without Guessing

Start from the workload, not the GPU.

Step 1: Define the workload. Model family and size, target context length, target concurrency, and latency budget. These four numbers determine everything else.

Step 2: Derive a memory floor. Weights at your chosen quantization, plus KV cache at target context and concurrency, plus runtime overhead. For the KV cache, estimate from the model's layer count, attention head dimensions, and bytes per element, then multiply by context length and concurrent requests. The exact formula varies by architecture and runtime, so use a runtime-specific calculator or profile the actual allocation. Add headroom — 20% is a reasonable starting point — for fragmentation and spikes, but treat that figure as a heuristic, not a portable guarantee.

Step 3: Match the runtime to the hardware class.

• CPU-only: viable for small models at low concurrency. Slow but universal.
• Integrated or unified memory: good for single-user interactive work. Higher capacity ceiling, lower bandwidth.
• Discrete accelerators: best for sustained throughput. Hard VRAM ceiling, higher bandwidth.
• NPU paths: battery-efficient sustained inference, typically for smaller models. Windows ML provides a unified framework for accessing NPUs, GPUs, and CPUs through execution providers.

Step 4: Prototype on the smallest configuration that can answer your real question. Then scale only the dimension that actually binds. If decode is bandwidth-bound, more VRAM capacity does not help. If you are hitting out-of-memory at long context, more bandwidth does not help.

Failure modes to expect: out-of-memory at long context, thermal throttling under sustained load, driver and runtime version drift, and format incompatibility between your chosen quantization and your serving stack.

The Binding-Constraint Decision Rule

The sections above describe several constraints. The practical question is which one should end the evaluation early.

Test them in order:

1. Fit. Do the weights, KV cache at target context and concurrency, and runtime overhead fit in available memory? If not, stop. Quantize, shrink the model, or move to hosted.
2. Latency and throughput. Does the configuration meet your TTFT and tokens-per-second targets at your real concurrency? If not, stop. More capacity will not fix a bandwidth wall.
3. Task quality. Does the quantized model clear your evaluation bar on your actual workload? If not, stop. A model that fits but fails your task does not fit your problem.
4. Privacy. Can you enumerate every outbound network call and confirm the boundary holds? If not, you have a hybrid system, not a local one — decide whether that is acceptable.
5. Fully loaded cost. Does the local option beat hosted when hardware, power, maintenance, and engineering time are all counted? If not, stop.

Failure at any hard requirement is a stop or hybrid-routing decision, not an invitation to keep upgrading hardware. The binding constraint is the one that fails first.

When Local Loses

Local inference is a capability trade, not a moral position. Choose it when the constraint it removes costs more than the capability it gives up.

Local is the wrong default when:

• The task needs frontier-level reasoning that small models cannot match.
• You need very long context or high concurrency on a budget.
• The team cannot absorb operational ownership. An unmaintained local stack degrades into a stale model on a hot machine.
• Privacy requirements are already satisfied by a contractual and architectural boundary you control.

Local is the right default when:

• Volume is high and sustained, making the amortized hardware cost competitive.
• Latency tolerance is generous enough for smaller models.
• Privacy or offline operation is a hard requirement.
• A small model already clears the accuracy bar on your evaluation set.

What to Learn and Build Next

Build a measurement harness before building a deployment. A small task-specific evaluation set, a latency and throughput script, and a memory profiler run at your target context length will tell you more than any benchmark table.

Learn the stack in dependency order: quantization formats first, then serving runtimes, then KV cache and batching behavior, then routing and evaluation. Each layer constrains the ones above it.

Run one narrow experiment that forces reality to answer: same model, two quantization levels, your own evaluation set, measured tokens per second and accuracy delta. That experiment will teach you more about your actual constraints than any amount of hardware research.

Track the signals that will change this decision: memory capacity and bandwidth on consumer accelerators, quality of small open-weight models at long context, and maturity of local routing and agent-harness tooling.

Keep the decision reversible. Design the application so the inference endpoint is swappable between local and hosted without rewriting the product. The constraint that makes local attractive today may not be the constraint that governs tomorrow.

The leverage question is not whether local inference is cheaper. It is what owning the inference layer lets you do that renting it does not — and whether that capability is worth the maintenance bill.