How Much GPU VRAM Do You Need for LLM Inference?

A useful inference memory budget includes resident weights, request state, runtime allocations, and transient peaks. A checkpoint that loads successfully is only the first milestone. The GPU must also survive the largest admitted prompt, its generated output, and simultaneous requests.

1. Calculate a raw-weight lower bound

Use weight bytes = total parameters × bits per weight / 8. With exactly 8 billion parameters, FP16 or BF16 needs 16 billion bytes (14.9 GiB), ideal 8-bit storage needs 8 billion bytes (7.45 GiB), and ideal 4-bit storage needs 4 billion bytes (3.73 GiB). GiB means 1,073,741,824 bytes. Round only after calculating.

These numbers exclude quantization scales, zero points, alignment, tensors retained at higher precision, and runtime workspaces. Artifact file size can also include metadata and is not a measurement of device allocation. Use total parameters for an MoE model unless the chosen runtime explicitly offloads or shards experts; active parameters alone undercount resident storage.

2. Estimate Transformer KV cache

For ordinary full-attention decoder layers with equal key/value head dimensions, a useful unsharded estimate is:

KV bytes = 2 × layers × KV heads × head dimension
           × bytes per cache element × total resident tokens

The factor two covers keys and values. Use the number of KV heads, not query heads, for grouped-query attention. Total resident tokens is the sum of retained prompt and generated tokens across active sequences. A conservative screen assumes every admitted sequence reaches its configured total length.

Illustrative architecture: 32 full-attention layers, 8 KV heads, head dimension 128, and two-byte cache elements. That is 131,072 bytes (128 KiB) per token. One 8,192-token sequence needs 1 GiB; four such sequences need 4 GiB. If instead there are 32 KV heads, the same example needs four times as much cache. These are hypothetical configurations, not specifications for every 8B model.

3. Add the runtime and a measured margin

Combining the illustrative 8B BF16 weights (14.9 GiB), four 8,192-token caches (4 GiB), and an assumed 3 GiB of other allocations gives 21.9 GiB. If you choose to keep 15% of device-visible capacity unused as an initial policy, the screen becomes 21.9 / 0.85 ≈ 25.8 GiB. Neither the 3 GiB overhead nor the 15% policy is a universal requirement. Replace them with observations from your actual stack.

Account for CUDA context, graph capture, attention workspaces, prefill activations, sampling buffers, and any draft model used for speculative decoding. Startup conversion can briefly require more memory than steady serving. Reserve CPU RAM and disk for loading as well; moving work off the GPU does not make it free.

Where the formula stops working

Sliding-window layers may stop growing their caches; static caches can reserve more than current token use; paged caches allocate in blocks; prefix sharing can reduce duplication but should not be assumed for arbitrary traffic. Beam search adds active hypotheses. Multimodal models can add encoder allocations and image/audio token state. Hybrid Mamba/attention models require state-space cache accounting as well. Consult the framework's cache documentation and the exact model configuration.

Measure in the process that performs inference

For PyTorch, place these calls around your existing model load or representative inference phase. Synchronization waits for queued GPU work; peak counters apply to the current process and selected device:

torch.cuda.synchronize()
torch.cuda.reset_peak_memory_stats()
# Run the existing model-load or inference phase here.
torch.cuda.synchronize()
print("peak tensor GiB:", torch.cuda.max_memory_allocated() / 2**30)
print("peak reserved GiB:", torch.cuda.max_memory_reserved() / 2**30)

Reserved memory already includes allocated tensors: do not add the two counters. NVIDIA SMI can show additional device usage outside this allocator. PyTorch memory documentation explains why emptying unused cache cannot free live tensors.

Turn the estimate into an admission policy

Test short, maximum-length, and mixed concurrent requests. If OOM occurs only during long generation, lower total context or active sequences before assuming a driver fault. If it occurs during loading, inspect artifact representation and initialization peaks. Cache preemption or repeated recomputation under load can indicate insufficient capacity even without OOM. Use the resulting limits in your server and document them alongside the GPU selection decision. Compare those requirements with live offerings, then use instance controls to complete the evaluation lifecycle.