GPU Benchmarking Basics: Synthetic and Workload Benchmarks

A GPU benchmark should answer one decision and provide enough detail to repeat it. A matrix-multiplication score tests one compute path; an inference benchmark tests a model and runtime; an endpoint test also includes scheduling and network overhead. Keep those measurement boundaries explicit.

Define success before collecting numbers

For interactive LLMs, measure time to first token, inter-token latency, end-to-end completion time, and errors. Time to first token includes queueing if timing begins when the client submits a request. Aggregate output tokens per second measures total completed output divided by wall time; it is not the sum of each request's reported generation speed. Count input and output tokens separately.

For embeddings, use documents or useful tokens per second at the real length distribution. For image generation, fix resolution, steps, guidance, and batch size. For training, include forward, backward, and optimizer work in a step-time result. A service benchmark should count failed and timed-out requests rather than silently removing them.

Record a reproducibility manifest

• GPU model, exact variant, device count, visible VRAM, topology, and host CPU/RAM.
• Driver, framework, CUDA runtime, extension versions, and container digest if used.
• Model and tokenizer revisions, weight format, cache dtype, and prompt template.
• Input/output lengths, batch or arrival schedule, active concurrency, sampling, and cache policy.
• Warm-up procedure, timed duration, request count, timeout rules, and background activity.

Use public or synthetic prompts and retain the benchmark script and configuration. Seed random inputs where meaningful, but do not promise bit-identical GPU execution across software versions. Record precision controls because a change in matrix-multiplication mode can alter both time and numerical behavior.

A small timing demonstration

This bounded PyTorch example allocates three 1,024-square FP16 tensors and times 20 matrix multiplications after five warm-ups. It requires an existing CUDA-capable PyTorch environment and briefly uses the GPU. Run it only on your evaluation device. It is a timing-method demonstration, not a GPU ranking or model benchmark.

import statistics
import torch

assert torch.cuda.is_available(), "CUDA is unavailable in this environment"
torch.manual_seed(0)
a = torch.randn((1024, 1024), device="cuda", dtype=torch.float16)
b = torch.randn_like(a)
c = torch.empty_like(a)
samples_ms = []
with torch.inference_mode():
    for _ in range(5):
        torch.mm(a, b, out=c)
    torch.cuda.synchronize()
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    for _ in range(20):
        start.record()
        torch.mm(a, b, out=c)
        end.record()
        end.synchronize()
        samples_ms.append(start.elapsed_time(end))
print("samples:", len(samples_ms))
print("median ms:", statistics.median(samples_ms))
print("min/max ms:", min(samples_ms), max(samples_ms))

GPU execution is asynchronous: timing only the Python call with a CPU stopwatch can measure enqueueing rather than completion. PyTorch's CUDA timing documentation explains synchronization and events. This example excludes allocation and host-to-device transfer, and serializes samples. It therefore says little about a pipelined application or network service.

Design an inference experiment

First establish one-request correctness and record cold startup separately. Warm the runtime, then test short prompts, long prompts, and a mixture. Increase concurrency gradually while keeping output limits controlled. A closed-loop client that waits for one response before sending another hides some queueing behavior; an arrival-rate test can expose overload. Bound its duration and request count and stop when errors or resource pressure exceed your trial limits.

Measure prefix-cache hits and misses separately. Repeating the same prompt can make a cache-enabled server look faster than normal traffic. Count actual generated tokens because early stopping changes work. Keep quality settings fixed; a faster model that produces unusable output is not a performance improvement for the application.

Report distributions and costs honestly

Include sample count, median and tail latency, completed throughput, error rate, and peak device memory. A p99 from a tiny sample is unstable; use enough observations and show run-to-run variation rather than false precision. Report the best sustainable operating point that still meets the latency and quality targets, alongside an equal-workload comparison.

For a dedicated instance, divide the actual billed evaluation cost by successful completed work, including startup and idle time in the stated window. Do not present a changing hourly catalog rate as a permanent benchmark constant. Use billing documentation to interpret the charge and current offerings for the exact machine.

If results regress, compare the manifest before tuning: changed prompt lengths, CPU contention, offload, cache policy, or kernels can explain more than the GPU name. For a capacity bottleneck, use the memory worksheet; for scaling, use the communication-aware parallelism guide. Preserve raw observations and distinguish measured results from hardware specifications.