GPU Performance Degradation Troubleshooting

A measurement-first workflow for slower inference, lower utilization, memory pressure, and changing throughput.

By Hostnot GPU · 9/6/2026

Start a performance investigation with one measurable regression: the same training step takes longer, image throughput falls, or request tail latency rises. GPU utilization alone is not an objective, and “100% busy” does not mean the workload is efficient.

Create a comparable baseline

Record model/code revision, input shape, precision, context and output length, batch or concurrency, GPU count and memory, driver, framework, image digest, and storage location. Separate model download, load, compilation, and warm-up from steady execution. For inference, distinguish queue wait, time to first token, decode rate, and end-to-end latency. Compare equal work: a shorter answer can make an apparent throughput improvement misleading.

Observe GPU and host together

nvidia-smi --query-gpu=timestamp,index,utilization.gpu,memory.used,power.draw,temperature.gpu,clocks.sm --format=csv -l 2
# In another terminal, where available:
vmstat 1 10

Stop the GPU loop with Ctrl+C. Unsupported fields should be omitted using the installed utility's help. vmstat can reveal CPU, runnable-process, paging, and I/O pressure; its first row is an average since boot, while subsequent rows cover the requested interval. Sampled GPU utilization can miss short bursts and does not distinguish compute-bound from memory-bound kernels.

Follow the bottleneck

Low GPU activity, busy CPU: investigate tokenization, decoding, augmentation, Python loops, or data-loader workers. More workers can oversubscribe CPUs or exhaust RAM.
Low activity with storage wait: compare local cached data with the normal source and inspect read throughput. Do not buy a larger GPU to fix model downloads.
High activity and long kernel time: profile representative operations, attention backend, precision, and memory traffic. High utilization does not identify the best optimization.
Latency worsens only under concurrency: inspect queue length and KV-cache demand. Set admission limits before increasing batches.
Clocks fall during a long run: correlate power/thermal event reasons; use thermal diagnosis if supported by evidence.
Multi-GPU slower than one: inspect synchronization, shard balance, and communication topology. Extra devices do not guarantee scaling.

Time CUDA operations correctly

CUDA launches are asynchronous. For an isolated PyTorch test, synchronize before starting and before stopping a host timer. This example times only a small matrix multiplication sequence; it is a timing demonstration, not a GPU benchmark or expected score.

import time
import torch

assert torch.cuda.is_available(), "CUDA is unavailable"
x = torch.randn(1024, 1024, device="cuda")
with torch.inference_mode():
    for _ in range(5):
        y = x @ x
    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(20):
        y = x @ x
    torch.cuda.synchronize()
print("seconds for 20 operations:", time.perf_counter() - start)

Do not insert synchronization into every operation of the production workload: it changes overlap and performance. For deeper analysis, use a short PyTorch Profiler or Nsight Systems trace with the real workload. Tracing adds overhead and may record tensor metadata, paths, or user content; restrict access and retention.

Prove the improvement and its cost

Change one variable, repeat enough warm measurements to report median and tail behavior, and check correctness and errors. Retain the baseline configuration for rollback. Report completed work per billed interval, including retries and idle setup time where relevant; hourly price alone cannot establish cost efficiency. Avoid upgrading drivers or raising power limits on a managed host in response to an unprofiled slowdown.

If escalation is needed, provide the fixed input shape, safe reproduction, before/after versions, and timeline. Next: verify distributed execution or design a reproducible benchmark.