How to Choose a GPU for LLM Inference

Choosing a GPU for LLM inference is an acceptance-test exercise. First define what the service must deliver; then remove configurations that fail memory, quality, or software constraints. Benchmark only the remaining candidates. This avoids optimizing tokens per second for a model that cannot answer your users' actual requests.

Write the deployment contract

Record an exact model repository and revision, tokenizer and chat template, license, weight format, cache dtype, maximum input tokens, maximum new tokens, and number of simultaneously active sequences. Include retrieval passages, system instructions, and tool definitions in the input budget. Define a latency target and error budget before testing, plus a separate quality test such as extraction accuracy or valid structured responses.

An illustrative contract might allow 4,096 input tokens and 512 output tokens with four active sequences. That is a 4,608-token per-sequence planning limit, not a claim that every model or server uses those settings. A queue of 100 waiting requests is different from 100 sequences whose caches are resident on the GPU.

Gate 1: memory feasibility

Estimate weights from total parameters and stored bytes, add architecture-specific cache, then add measured runtime and transient allocations. An 8-billion-parameter BF16 model starts at 16 billion bytes, approximately 14.9 GiB, before everything else. Quantized weights can lower this term, but do not automatically quantize the cache. Use the VRAM calculation guide rather than treating parameter count as a complete requirement.

For mixture-of-experts models, active parameters describe part of the per-token computation, not all weights that need storage. For hybrid state-space architectures, a standard Transformer cache formula is incomplete. Use the model's own configuration and the serving engine's memory report.

Gate 2: the software path

Check GPU architecture support in the runtime release, the quantization backend, attention kernels, and any custom extensions. NVIDIA's compute-capability table distinguishes A100, Hopper, and GeForce Blackwell; a CUDA label alone does not establish kernel compatibility. Pin the tested image and artifact revision. Verify correct stopping, chat formatting, and output parsing before interpreting performance.

Gate 3: latency under arrivals

Measure prefill and decode separately. Prefill processes the prompt and contributes to time to first token; decode produces subsequent tokens. Large prefill batches can delay ongoing generation. A larger batch can improve aggregate throughput while worsening per-user latency. Test short and long prompts together, not just identical requests launched after each predecessor finishes.

1. Run one short request to validate loading and correctness.
2. Run the longest admitted prompt and reserve room for its output.
3. Increase active concurrency in bounded steps while recording tail latency, cache pressure, and failures.
4. Introduce a brief arrival burst and verify that admission control or the queue behaves predictably.
5. Repeat after a fresh start to separate warm performance from loading and compilation.

Choose a topology only when it solves a problem

If the model fits on one device, independent replicas may be a better throughput option than splitting every request. If it does not fit, compare a larger-memory card, an acceptable quantization, and supported model parallelism. vLLM's scaling guide explains these placements. Communication and replicated buffers mean two cards rarely behave like one card with twice the memory.

Make the final decision with operating cost

For a dedicated evaluation, divide the actual total billed cost by successful requests or accepted output tokens, while enforcing the latency and quality gates. Include failed attempts and unused time in that cost. Record the exact offering rather than a family name. Review current GPU configurations and usage and budget guidance. For a compatible hosted model, compare the Serverless API workflow against the operational work of maintaining a dedicated runtime.