Tensor Parallelism Explained for Multi-GPU Inference

Tensor parallelism splits selected operations within a model across GPUs. It can make a model fit when one GPU cannot hold it, but it adds communication during inference. It is a model-execution strategy, not automatic memory pooling and not a guarantee that two GPUs serve a request twice as fast.

What is actually split?

For a matrix multiplication, a runtime can divide weight columns so each GPU computes part of the output, or divide rows and combine partial results. Transformer implementations coordinate these partitions across attention and feed-forward layers. Collective operations such as all-reduce or all-gather exchange the necessary results. The exact partitioning and supported tensor-parallel sizes depend on the architecture and engine.

Some tensors remain replicated. KV heads may be partitioned or replicated depending on the model and parallel size. Vocabulary padding, workspaces, and rank-local allocations also consume memory. Thus a rough weight estimate divided by GPU count is a starting point, not a per-rank memory guarantee.

Distinguish the alternatives

• Independent replicas/data parallel serving: each replica holds a model and serves different requests. Useful when one-device fit is already solved and throughput is the goal.
• Tensor parallelism: ranks collaborate within layers for the same model execution. Useful for capacity, but sensitive to communication latency and bandwidth.
• Pipeline parallelism: different stages hold different layers. It can accommodate some uneven partitions, but scheduling and pipeline bubbles affect utilization.
• Expert parallelism: routes MoE work to partitioned experts. This is model-specific and introduces its own routing communication.

For example, 60 GiB of raw weights across two ranks suggests roughly 30 GiB per rank if perfectly split. On two 40GB-class cards, replicated allocations and request state can still exceed the remaining budget. Compare this with a single larger-memory device before assuming the two-card design is simpler.

Inspect topology before launching workers

nvidia-smi --query-gpu=index,name,memory.total --format=csv
nvidia-smi topo -m

These read-only commands show visible devices and topology. The matrix legend distinguishes NVLink paths from PCIe/host paths; it does not certify achieved collective bandwidth. Container visibility, peer-access restrictions, and runtime transport selection still matter. Review NVIDIA's topology documentation and keep results with the deployment manifest.

A conservative single-node trial

In an already prepared environment, inspect vllm serve --help and the release's model-support documentation. The following is an illustrative two-visible-GPU command using an already downloaded, reviewed model directory. Replace the path and confirm that this model supports TP=2. It starts a server and allocates GPU memory; run it on an evaluation instance, not alongside a production workload.

vllm serve /path/to/reviewed-model --tensor-parallel-size 2 --max-model-len 4096 --max-num-seqs 1 --host 127.0.0.1

The context and concurrency are deliberately bounded trial settings, not recommended limits for every model. Confirm two workers initialize, both have intended device placement, and the engine reports a usable cache budget. Run one small request before increasing limits. Stop the trial server after collecting results. The official vLLM scaling guide explains single-node and multi-node placements.

Diagnose common failures

Unsupported partition: inspect the model's head counts and engine constraints; do not assume any GPU count divides every architecture. One rank OOM: compare allocations per rank and identify replicated or uneven tensors. Initialization hang: check device visibility, identical package versions, shared-memory requirements, and the first worker error before changing network settings.

Slower generation: compare one-card performance where feasible, and inspect time in collectives. Adding ranks may save capacity while increasing per-token latency. Do not blindly disable communication safety checks or expose distributed control ports publicly. Multi-node deployment needs an explicit private-network design and is not implied by a two-GPU offer.

Use repeatable request-shape benchmarks and per-device memory accounting to decide whether the tradeoff works. Review GPU cloud topology guidance and the exact current machine configuration before scaling.