When One GPU Isn't Enough

Sooner or later one GPU isn’t enough, for one of three reasons. The model doesn’t fit: a 70B model in 16-bit needs 140 GB, and an H100 holds 80. It isn’t fast enough: even when it fits, one GPU’s memory bandwidth caps how quickly tokens appear. Or there are simply too many users.

The last case is the easiest: run more copies of the model (“replicas”) and let a load balancer share the traffic between them. The first two need the model itself split up, and there are two main ways to cut it.

Tensor parallelism cuts every layer lengthwise. Each GPU holds a slice of every weight matrix and computes its share of every layer, so all of them work on every token at once. Each GPU reads only its slice from memory, so a single user’s tokens arrive faster. The cost is chatter: after every layer, the GPUs must swap partial results before carrying on, so they need extremely fast links between them.

Pipeline parallelism cuts the stack of layers into blocks: GPU 1 runs layers 1–20, GPU 2 runs 21–40, and so on, passing each token along like an assembly line. Much less chatter, but one user’s token still visits every stage in turn, so it isn’t faster for that user. It pays off when many requests keep every stage busy at once.

That fast link matters a lot. Inside one server, usually eight GPUs, NVIDIA’s NVLink connects them at hundreds of gigabytes per second, ideal for tensor parallelism. Between servers, links are much slower. So a common recipe is tensor parallelism inside a server, pipeline parallelism across servers, and replicas on top for traffic. Mixture-of-experts models add a fourth option, expert parallelism: different GPUs hold different experts, and tokens travel to wherever their chosen experts live.