FP16 vs BF16 vs FP8 for AI Workloads

FP16, BF16, and FP8 trade numerical precision and range for smaller representations. Three settings must be distinguished: how weights are stored, the format used by a kernel, and the precision used to accumulate results. A model described as FP8 may still keep sensitive operations and caches in higher precision.

FP16 versus BF16: equal storage, different numbers

FP16 uses one sign bit, five exponent bits, and ten fraction bits. BF16 uses one sign bit, eight exponent bits, and seven fraction bits. Both consume two bytes per element. BF16 has an exponent range comparable to FP32, while FP16 provides more fraction precision near a given scale. Neither is universally more accurate for every operation.

A practical consequence is that FP16 cannot represent finite values above 65,504. Converting a BF16-trained checkpoint to FP16 can overflow if values or intermediates exceed that range. BF16's wider range does not eliminate rounding error, and reductions can still be numerically sensitive. Training may use gradient scaling with FP16 to protect small gradients; that is not an inference-memory optimization.

FP8 is a family, not a universal switch

E4M3 and E5M2 allocate exponent and fraction bits differently: four and three, or five and two, plus a sign bit. Encoding variants differ in special-value behavior. Scaling maps a tensor's values into the usable range, and the granularity of those scales matters. An FP8 recipe includes the encoding, scale handling, eligible operations, and higher-precision fallbacks. NVIDIA's versioned Transformer Engine FP8 primer explains these representations and scaling choices.

NVIDIA's H100 specifications include FP8 Tensor Core support. That does not mean an arbitrary checkpoint cast to an eight-bit dtype runs correctly or faster. A100's advertised BF16/FP16 path should not be confused with Hopper's native FP8 Tensor Core path. An engine may store compressed weights and compute after conversion; storage support and native arithmetic are different claims.

What changes in memory?

For exactly 8 billion weights, an all-16-bit representation is 16 billion bytes, about 14.9 GiB. An ideal all-8-bit representation is 8 billion bytes, about 7.45 GiB. These are raw-weight calculations. Scales, unconverted layers, temporary conversion buffers, graph pools, and cache remain additional costs. BF16 instead of FP16 offers no raw-weight storage saving.

KV-cache dtype is a separate setting. Reducing weight precision may leave long-context memory nearly unchanged if the cache dominates. Conversely, reducing cache precision can introduce its own quality and latency tradeoffs. See Transformers cache strategies and the quantization guide.

A small numerical diagnostic

In an existing PyTorch environment, this CPU-only example inspects representable ranges. It loads no checkpoint and performs no GPU benchmark:

import torch

for dtype in (torch.float16, torch.bfloat16, torch.float32):
    info = torch.finfo(dtype)
    print(dtype, "max:", info.max, "epsilon:", info.eps)
    print(torch.tensor([1.001, 70000.0], dtype=dtype))

Expect FP16's 70,000 value to become infinity, while BF16 remains finite but rounded. Epsilon describes spacing near one, not the error of an entire model. This diagnostic demonstrates representation only; it does not prove that your GPU has an optimized kernel for that dtype.

Debug a precision regression

Keep a known-good higher-precision run with the same prompts, tokenizer, sampling, and model revision. If a reduced-precision run produces NaNs, endless repetition, or invalid structured answers, first verify the intended backend and scaling recipe. Compare prefill and decode outputs on a small case. Changing attention or reduction kernels can also change numerical behavior; PyTorch's CUDA semantics documents reduced-precision reductions and accumulation controls.

Adopt a format only after it passes task quality, peak-memory, and latency tests. Use a reproducible benchmark and compare available hardware using the exact precision path you tested, rather than a peak-FLOPS label.