RTX 5090 for AI: VRAM, Inference and Workload Guide

The RTX 5090 is a 32GB GDDR7, Blackwell-generation GeForce GPU according to NVIDIA's specifications. For AI, its useful envelope is set by model memory, available software kernels, and the host system. Start by proving those three conditions rather than extrapolating from gaming performance or advertised AI TOPS.

Which workloads are sensible candidates?

Bounded-context language-model inference, embeddings, image generation, and development experiments are candidates when their complete working sets fit. They are not interchangeable benchmarks. Embeddings may be limited by tokenization; image generation may peak in the decoder; video generation can grow sharply with frame count and spatial resolution. Measure the entire pipeline you intend to keep resident.

For LLM screening, exactly 8 billion BF16 parameters require about 14.9 GiB of raw weights; exactly 14 billion require 26.1 GiB. The latter leaves much less room for cache and workspaces on a 32GB-class device. These are idealized parameter calculations, not claims that a particular model fits at a particular context. A quantized 32B model has another footprint again: even ideal four-bit weights start around 14.9 GiB before scales and runtime state.

Blackwell support is more specific than “CUDA installed”

NVIDIA lists RTX 5090 at compute capability 12.0. Do not assume a binary built for an A100 or a data-center Blackwell target contains compatible code for it. Verify the framework release and every compiled extension used by the model, including attention, quantization, and fused operators.

The CUDA version shown by NVIDIA SMI describes driver capability, not necessarily the toolkit or runtime used by a Python environment. Run this small inventory check inside the intended environment; it imports installed PyTorch but downloads no model:

import torch

print("PyTorch:", torch.__version__, "built with CUDA:", torch.version.cuda)
print("CUDA available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("GPU:", torch.cuda.get_device_name(0))
    print("capability:", torch.cuda.get_device_capability(0))
    print("compiled targets:", torch.cuda.get_arch_list())
    print("device GiB:", torch.cuda.get_device_properties(0).total_memory / 2**30)

A missing target in the list is a compatibility clue, not a complete diagnosis: PTX JIT paths and third-party extensions complicate the picture. Follow the runtime's documented support matrix and test a small representative operation before a full model. An error such as “no kernel image is available” points toward binary/architecture compatibility rather than insufficient VRAM.

Memory limits need an explicit policy

Count prompt plus generated tokens and bound active sequences. If the engine reserves a large cache at startup, high memory usage can be expected; use its cache occupancy and preemption metrics to distinguish reservation from pressure. A successful short answer does not prove the maximum advertised model context fits. Calculate and measure cache demand before opening access to longer requests.

CPU offload can enable a larger working set but changes latency and host-RAM requirements. Record offload explicitly in results. A desktop using the GPU for display can also have a different free-memory budget than a dedicated environment. Neither a second graphics card nor more system RAM transparently expands a single CUDA allocation.

A practical evaluation ladder

1. Inventory device-visible memory, driver, framework, and extension versions.
2. Load the pinned artifact and verify intended GPU placement and quantization.
3. Run one normal request, then the largest supported request you intend to admit.
4. Increase batch or active concurrency one step at a time while watching memory and tail latency.
5. Repeat after warm-up and after a fresh process start, reporting both conditions.

If throughput drops during a longer run, inspect reported temperature, power, and clocks as well as CPU and disk activity. Do not treat a fixed temperature or board-power number as a universal operating target. Avoid changing host power limits or resetting a shared GPU as a benchmark shortcut. NVIDIA SMI documentation distinguishes queries from device-modification commands.

For larger-memory requirements, compare RTX 5090 and A100 deployment tradeoffs. For the named hybrid model, read the Nemotron-specific artifact review. Use live RTX 5090 offerings and the cloud quickstart to select and verify an actual environment.