LLM Inference Quantization: 4-bit and 8-bit Guide

Quantization can make an otherwise impractical model fit, but the choice is an artifact-and-kernel decision rather than a bit-count contest. A smaller checkpoint is useful only if the runtime supports it, its outputs meet the task's requirements, and the complete service improves at the intended request shape.

Identify what is actually quantized

Weight-only quantization compresses stored parameters while computation may use higher-precision activations. W4A16 means four-bit weights and 16-bit activations; it does not mean every operation runs as native four-bit arithmetic. Weight-and-activation quantization changes more of the compute path and needs appropriate kernels and scaling. KV-cache quantization is independent of either choice and affects request-state memory.

A hypothetical 13-billion-parameter model occupies 26 billion raw weight bytes in BF16 (24.2 GiB). Perfectly packed four-bit weights would occupy 6.5 billion bytes (6.05 GiB). Real allocations include scales, zero points where used, padding, unquantized tensors, workspaces, and cache. The ideal 4× reduction applies only to this raw-weight term, not the entire process.

Do not confuse methods with containers

• AWQ and GPTQ-family artifacts: quantization approaches with particular tensor layouts and metadata. Confirm the exact backend, group size, and supported GPU architecture.
• bitsandbytes: a loading/integration path with its own supported formats and compute-dtype choices. Loading a model in four bits is not the same operation as loading an already-converted AWQ checkpoint.
• GGUF: a file format that can contain multiple tensor quantization types. A Q4 variant's actual footprint depends on the recipe, not merely the letter Q.
• FP8 or NVFP4: floating-point representations and associated recipes. Hardware support, scaling, and the selected kernel determine whether a compressed artifact uses native low-precision arithmetic.

Consult the current Transformers quantization overview and your engine's release documentation. Compatibility can differ by architecture and package version. Renaming files or adding a generic quantization flag cannot convert an incompatible artifact.

Choose a candidate in four steps

1. Establish a reference using the publisher's supported higher-precision model, tokenizer, and template. Keep this run's configuration and task results.
2. Prefer a documented artifact with traceable source revision and conversion settings. If converting yourself, keep calibration data separate from the evaluation set and representative of your workload.
3. Confirm startup logs identify the intended quantization backend and placement. Check for CPU offload, fallback kernels, or unexpected dequantized weights.
4. Evaluate memory, quality, and latency with the same prompt set before increasing concurrency.

Evaluate the failures your application cares about

For extraction, compare field accuracy and valid-schema rate. For coding, run the same tests on generated code in an appropriate sandbox. For retrieval-assisted answers, evaluate grounding and long-context behavior. Include multilingual or numerical tasks if they matter to users. A few fluent responses are not a quality evaluation, and small score changes may be sampling noise.

Keep generation settings equal and report sample size. If exact strings differ, inspect task success rather than requiring bit-identical output across precisions. If quality fails, try a less aggressive representation or preserve sensitive modules at higher precision using a supported recipe. Do not hide the difference by changing the model or prompt template in the same experiment.

Why four bits can be slower

Dequantization work and poorly matched kernels can offset reduced weight traffic. Small batches may use a different kernel than large ones. CPU offload adds transfers. Cache-heavy long contexts may barely benefit from smaller weights. Test both time to first token and sustained generation; compare the same concurrency before using newly freed memory for larger batches.

For OOM, identify the phase: conversion/loading, graph capture, prefill, or decode. A smaller artifact may still require a large conversion workspace. Use the VRAM worksheet and cache strategy documentation to identify the term that needs reducing. Choose a current GPU offering only after confirming the kernel path, and use GPU cloud documentation to prepare the runtime.