H100 vs H200 for LLM inference: memory, throughput and cost

Choosing between H100 and H200 for large language model inference is not simply a contest over peak speed. The practical questions are whether the model and its working state fit in memory, how much bandwidth serving requires, and whether the added throughput justifies the instance rate.

For the common SXM configurations, H100 offers 80 GB of GPU memory and about 3.35 TB/s of memory bandwidth. H200 offers 141 GB and about 4.8 TB/s. Those are useful reference points, not a guarantee about every cloud instance: GPU configurations and available capacity can vary. Check the exact offering rather than assuming a GPU name tells the whole story.

Memory capacity changes what fits

Model weights are only part of inference memory use. Runtime overhead, context length, concurrent requests and the key-value cache all consume VRAM. A model that appears to fit based on weight size alone may leave too little room for useful concurrency or long prompts.

H200’s 141 GB can make a difference when a workload approaches the practical limit of an 80 GB H100. More room may allow a model to run on one GPU, accommodate longer contexts, or serve more simultaneous requests, depending on the model and serving setup. Avoiding a multi-GPU arrangement can simplify deployment, but it does not automatically make a single H200 faster or cheaper for every workload.

If the model already fits comfortably on H100, additional capacity may not improve results. In that case, request rate, batching, precision, software implementation and latency targets can matter more than unused VRAM.

Bandwidth matters, but it is not a throughput forecast

Autoregressive generation repeatedly reads model weights as it produces tokens. Memory bandwidth can therefore be important, particularly for workloads where generation is limited by moving data rather than by computation. H200’s higher bandwidth gives it a hardware advantage in that respect.

Still, a bandwidth ratio is not a reliable prediction of tokens per second. Actual performance depends on factors such as model architecture, prompt and output lengths, batch size, quantization, serving software and how much work the GPU can run in parallel. Prefill, which processes an input prompt, can behave differently from token-by-token generation. Measure both if both matter to your application.

Test with the model, request mix and concurrency you expect in production. Record time to first token, output tokens per second, total request latency and the number of requests handled at your chosen latency limit. A single peak-throughput number can hide a poor fit for interactive service.

Compare cost per useful output

There is no universal answer to “H200 vs H100 speed” that also settles value. The comparison needs the actual rate for each available instance and a benchmark under the same conditions. A practical measure is cost per million generated tokens: instance cost per hour divided by measured output tokens per hour, then scaled to a million tokens. For interactive workloads, include latency and concurrency constraints; the cheapest token is not useful if it misses the service target.

Cloud pricing may vary with the selected offering and current capacity. Hostnot GPU says its rates are calculated server-side and directs customers to review the displayed rate before deployment. Its catalog can be filtered by GPU model, memory, region and capacity class. The service lists 91 offerings across 20 GPU types and eight regions, but those totals do not establish that a specific H100 or H200 configuration is currently available. Check the live catalog and compare the exact configurations and rates shown.

Which tier suits which workload?

H100 is a reasonable starting point when the model fits within its usable memory, a benchmark meets the throughput target, and its instance rate produces a better cost for your request mix. It can also suit teams that want to test a smaller configuration before committing to more capacity.

H200 is worth testing when memory pressure limits model size, context length or concurrency, or when bandwidth-sensitive generation is the bottleneck. Its larger memory pool may change the deployment shape as much as the speed: compare a single-GPU H200 with the actual alternative you would run, not just with a specification sheet.

If you need a persistent environment, custom runtime or direct SSH access, an on-demand GPU instance is the relevant category to evaluate. Hostnot GPU offers SSH-ready instances using registered public keys. For supported request-and-response workloads that do not require a full environment, its serverless inference option supports 50 models for chat, image or video requests; that is a different deployment path, not a like-for-like H100/H200 instance comparison.

Region can also affect end-user response time, so benchmark close to the users your service will support. The practical trade-off between catalog location and observed latency is covered in how to choose a GPU cloud region for inference.

Make the decision with a repeatable test

Use the same model, software, precision, request sample and latency target on both configurations. Track peak and steady-state behavior, then calculate cost from the current displayed rates. If one option cannot fit the required model or context in memory, remove it from the comparison. If both meet the target, choose based on measured cost and operational fit rather than the larger specification.

H200 offers a substantial memory-capacity and bandwidth increase over H100 SXM. Whether that translates into better GPU cloud inference value depends on the workload, the actual instance configuration and its rate. Run the benchmark before scaling the choice into production.