NVIDIA H100 vs H200: Which GPU Should You Choose?

H200's larger, faster memory can change which LLM workloads fit on one GPU and how many sequences remain resident. That is the central selection question. It does not establish a universal speedup over H100, especially when your bottleneck is compute, CPU preprocessing, or request scheduling.

Compare named configurations

NVIDIA's H100 specification table lists H100 SXM at 80GB and 3.35 TB/s memory bandwidth, and H100 NVL at 94GB and 3.9 TB/s per GPU. The 188GB figure used for a two-GPU H100 NVL pair is aggregate capacity, not one GPU's allocation space. The family name alone is insufficient to choose a memory budget.

The H200 specification table lists 141GB and 4.8 TB/s for both SXM and NVL variants, while their power and platform details differ. H200 uses HBM3e and remains a Hopper architecture GPU. Keep form factor, GPU count, and interconnect in the comparison record. Published bandwidth is a hardware ceiling, not measured token throughput.

Capacity can change the deployment shape

Consider a hypothetical 70-billion-parameter model whose raw weights are all BF16. At two bytes per parameter that is 140 billion bytes, about 130.4 GiB, before runtime buffers and request state. A “141GB” label therefore is not sufficient evidence that this model serves useful traffic on one H200. Measure device-visible memory and the exact checkpoint footprint in the same units.

An ideal eight-bit representation of that rounded parameter count starts at 65.2 GiB. That could substantially change the capacity screen, but scales, mixed-precision tensors, and the cache still need room. The useful question becomes whether an acceptable artifact plus the required context and concurrency fits, not whether the total parameter count sounds compatible with a product name.

When H200 is worth testing first

• Your H100 configuration runs out of cache capacity at the required context or active sequence count.
• A larger working set on one card could avoid offloading or an extra tensor-parallel worker.
• Profiling suggests memory traffic is limiting decode, and the software supports the same precision path on both candidates.

Test H100 first when the workload already fits and the decision is primarily cost at an acceptable latency. Test both for large batches: batching changes arithmetic intensity and can move the bottleneck. A manufacturer's comparison using different batch sizes is not an equal-request-latency result.

Separate prefill, decode, and communication

Long prompt processing can create compute and workspace pressure before the first token. Subsequent generation repeatedly accesses weights and retained state; at some batch sizes memory traffic is central. With multiple GPUs, collective communication adds another cost. Record all three phases rather than inferring the bottleneck from one utilization percentage.

Start with the same artifact, runtime, prompt distribution, precision, and active concurrency. Compare latency and memory. Then run a separate capacity test to find each system's sustainable concurrency under the same latency limit. Label these two experiments: equal workload and best acceptable operating point. Otherwise a larger batch can look like a free speedup while individual users wait longer.

Check the offered machine

nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv
nvidia-smi topo -m

Inventory queries help confirm device memory and visible topology but cannot certify an application communication path. Verify the serving engine's worker placement and interconnect logs. If additional GPUs do not improve the target metric, inspect communication time and replicated allocations before adding more. See tensor parallelism and benchmark reporting.

Use H100 offerings and H200 offerings for current configuration choices. Follow Marketplace guidance to review the complete machine and authorized rate; no location, inventory, or fixed price is implied by this hardware comparison.