Beyond One GPU: How LLM Inference Scales Across Clusters

Why one GPU stops being enough

A 70B model in FP16 needs about 140 GB just for weights. The largest single GPUs today carry 80–192 GB of HBM, and you still need room for the KV cache, which grows with every token of every concurrent request. Frontier MoE models like DeepSeek-V3/R1 (671B total parameters) do not fit on a single 8-GPU node in BF16 at all.

So serving LLMs at scale is a partitioning problem. You are always splitting one of four things across GPUs:

• Requests — Data Parallelism (more replicas, more throughput)
• Layers — Pipeline Parallelism (the model's depth is split)
• Matrices — Tensor Parallelism (each layer's math is split)
• Experts — Expert Parallelism (MoE experts live on different GPUs)

And increasingly, a fifth split: phases. Prefill/Decode disaggregation runs prompt processing and token generation on separate hardware pools.

Every choice trades memory against communication. The rule of thumb that drives everything below: the faster the interconnect, the finer-grained the split it can sustain. NVLink (hundreds of GB/s to 1.8 TB/s per GPU) handles tensor-level splits. InfiniBand/RoCE (tens of GB/s per link) handles layer-, expert- and phase-level splits. Plain Ethernet handles request-level splits.

Let's walk each strategy, then put it all together on Kubernetes.

1. Data Parallelism (DP) and FSDP

Data Parallelism is the simplest scale-out: put a full copy of the model on each GPU (or group of GPUs) and route different requests to each copy. Throughput scales almost linearly, and replicas don't talk to each other during a forward pass.

In training, PyTorch's DistributedDataParallel replicates the model and all-reduces gradients after each backward pass. In inference there are no gradients, so "DP" mostly means N independent replicas behind a load balancer. The catch: every replica must fit the whole model.

FSDP and ZeRO remove that constraint by sharding state. PyTorch FSDP shards parameters, gradients and optimizer state across data-parallel ranks, then all-gathers a layer's full weights just before it runs and frees them right after. DeepSpeed ZeRO defines three stages:

Stage	What is sharded	Memory saved	Matters for inference?
ZeRO-1	Optimizer state	~4x	No (no optimizer)
ZeRO-2	+ Gradients	~8x	No (no gradients)
ZeRO-3	+ Parameters	Linear in # of GPUs	Yes

ZeRO-Inference builds on stage 3: it keeps weights in CPU or NVMe memory and streams them into the GPU layer by layer. That lets a single GPU run a model far larger than its HBM, at the cost of latency. It shines for offline, throughput-oriented batch jobs, not chat.

import torch.distributed as dist
from torch.distributed.fsdp import fully_shard

dist.init_process_group("nccl")
model = build_model()
for block in model.layers:
    fully_shard(block)   # shard each transformer block
fully_shard(model)       # shard the root

In modern serving engines, DP has a new meaning. vLLM and SGLang support "DP attention": attention runs data-parallel (each rank has its own KV cache and batch) while the MoE layers run expert-parallel across the same GPUs. This avoids duplicating KV cache across TP ranks, which is critical for models like DeepSeek that use Multi-head Latent Attention.

Use DP when: the model already fits on one GPU or one node and you need more QPS. It's the first and cheapest lever.

2. Pipeline Parallelism (PP)

Pipeline Parallelism splits the model vertically: GPU 0 holds layers 0–19, GPU 1 holds layers 20–39, and so on. Activations flow from stage to stage like an assembly line. Only one tensor (the hidden state at the stage boundary) crosses the wire per micro-batch, so PP tolerates slower links and is the standard way to span multiple nodes.

The problem is the pipeline bubble: while stage 0 works on the first micro-batch, stages 1–3 sit idle. The fix is to cut the batch into micro-batches and keep every stage busy. DeepSpeed's pipeline engine popularized the 1F1B schedule (one forward, one backward, interleaved) for training. With p stages and m micro-batches, the idle fraction is roughly:

\text{bubble} \approx \frac{p - 1}{m + p - 1}

More micro-batches mean a smaller bubble. For inference there is no backward pass, so serving engines keep the pipeline full by having many independent requests in flight at once.

PyTorch now ships this natively in torch.distributed.pipelining: you split the model into stages and pick a schedule (ScheduleGPipe, Schedule1F1B, ScheduleInterleaved1F1B).

from torch.distributed.pipelining import pipeline, SplitPoint, ScheduleGPipe

pipe = pipeline(
    model,
    mb_args=(example_microbatch,),
    split_spec={"layers.20": SplitPoint.BEGINNING},
)
stage = pipe.build_stage(rank, device)
schedule = ScheduleGPipe(stage, n_microbatches=8)

In vLLM it's a single flag:

vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2   # 16 GPUs across 2 nodes

The inference trade-off: PP improves throughput and capacity, but it does not reduce per-token latency. Each token still traverses every layer sequentially, plus a network hop per stage boundary.

Use PP when: the model doesn't fit in one node and your inter-node link is too slow for tensor parallelism.

3. Tensor Parallelism (TP)

Tensor Parallelism splits each layer horizontally. Every GPU holds a slice of every weight matrix and computes part of every token. It's the only strategy here that cuts per-token latency, because all GPUs work on the same token at the same time.

The pattern comes from NVIDIA Megatron-LM. Take a transformer MLP, Y = GeLU(X·A)·B:

1. Split A by columns. Each GPU computes its slice of X·A independently. GeLU is element-wise, so no communication is needed yet.
2. Split B by rows. Each GPU produces a partial sum of the output.
3. One all-reduce combines the partial sums.

Attention works the same way: heads are split across GPUs (column-wise on Q/K/V, row-wise on the output projection). The result is two all-reduces per transformer layer, on every forward pass, for every token. That traffic is why TP lives inside a node on NVLink and rarely crosses InfiniBand.

PyTorch exposes this directly in torch.distributed.tensor.parallel:

from torch.distributed.device_mesh import init_device_mesh
from torch.distributed.tensor.parallel import (
    parallelize_module, ColwiseParallel, RowwiseParallel,
)

mesh = init_device_mesh("cuda", (8,))
for block in model.layers:
    parallelize_module(block, mesh, {
        "attn.q_proj": ColwiseParallel(),
        "attn.k_proj": ColwiseParallel(),
        "attn.v_proj": ColwiseParallel(),
        "attn.o_proj": RowwiseParallel(),
        "mlp.up_proj": ColwiseParallel(),
        "mlp.down_proj": RowwiseParallel(),
    })

Production runtimes ship fused, hand-tuned versions. TensorRT-LLM builds TP and PP into its engine with custom all-reduce kernels that overlap communication with compute. vLLM and SGLang expose it as --tensor-parallel-size / --tp.

Watch out: TP degree must divide the number of attention heads (and KV heads, for GQA models). Past TP=8, the all-reduce cost usually outweighs the gain, unless you're on a rack-scale NVLink domain like GB200 NVL72.

Use TP when: you need lower latency, or the model needs more memory than one GPU but fits in one NVLink-connected node.

4. Expert Parallelism (EP) for Mixture-of-Experts

Mixture-of-Experts models replace the dense MLP with many small "expert" MLPs and a router that sends each token to only a few of them. DeepSeek-V3/R1 has 256 routed experts per MoE layer and activates 8 per token, so only ~37B of its 671B parameters do work for any given token. Mixtral 8x7B activates 2 of 8.

That makes MoE cheap to compute but huge to store. Expert Parallelism places different experts on different GPUs, so each GPU holds only a fraction of the expert weights.

Each MoE layer then runs in four steps:

1. Route — the gating network picks the top-k experts for every token.
2. Dispatch (All-to-All) — each GPU sends its tokens to whichever GPUs host their chosen experts.
3. Compute — each GPU runs its local experts on the tokens it received.
4. Combine (All-to-All) — results travel back to the token's home GPU and are weighted by the router scores.

Those two All-to-All exchanges per layer are the whole game. DeepSpeed MoE covers expert partitioning and All-to-All tuning, and DeepSeek open-sourced DeepEP, a communication library built specifically for low-latency MoE dispatch/combine over NVLink and RDMA.

The operational headache is load imbalance. If the router favours a few "hot" experts, the GPUs hosting them become stragglers and everyone waits. Production systems answer with Expert Parallel Load Balancing (EPLB): replicating hot experts and rebalancing placement based on observed traffic.

vLLM runs sparse MoE models like Mixtral and DeepSeek-V3/R1 natively. A typical wide-EP deployment combines DP attention with EP for the expert layers:

vllm serve deepseek-ai/DeepSeek-V3 \
  --data-parallel-size 16 \
  --enable-expert-parallel

Use EP when: you serve MoE models. For large MoE models at scale it's not optional; it's how the experts fit and how the cluster stays busy.

5. Prefill/Decode (PD) Disaggregation

Every LLM request has two phases with opposite hardware profiles:

	Prefill	Decode
Work	Process the whole prompt in one pass	Generate one token at a time
Bottleneck	Compute (large matmuls)	Memory bandwidth (reading weights + KV cache)
Latency metric	Time to First Token (TTFT)	Time Per Output Token (TPOT / ITL)
Ideal parallelism	Higher TP, fewer large batches	Wide EP/DP, very large batches

When both phases share a GPU, they interfere. A 32K-token prompt arriving mid-stream stalls every in-flight decode, and users see their streaming text freeze. Chunked prefill softens this by slicing long prompts, but it can't make one GPU configuration optimal for both phases.

Disaggregation runs them on separate worker pools. A prefill worker builds the KV cache, ships it to a decode worker over RDMA/InfiniBand (or NVLink within a node), and the decode worker streams the tokens. Each pool is sized and parallelized for its own job, and you can scale them independently: add prefill nodes for long-context traffic, decode nodes for long outputs.

The cost is the KV transfer itself. For a large model, a long prompt's KV cache can be gigabytes, so this only works with fast fabric and a purpose-built transfer layer (NIXL, Mooncake, LMCache).

All three major engines support it:

• vLLM production-stack — separate prefill and decode deployments, with LMCache/NIXL moving the KV cache between them.
• SGLang — --disaggregation-mode prefill|decode workers behind a router, KV transfer over RDMA.
• TensorRT-LLM — disaggregated "context" and "generation" worker pools, also the backbone of NVIDIA Dynamo.

Use PD disaggregation when: you have strict TTFT and TPOT SLOs at the same time, long or highly variable prompts, and an RDMA-capable network. At small scale, chunked prefill on co-located workers is usually simpler and good enough.

6. Putting it together: multi-dimensional parallelism

Real deployments combine strategies, mapped onto the hardware hierarchy: TP inside a node (NVLink), PP or EP across nodes (InfiniBand), DP across replicas (anything). Total GPUs = TP × PP × DP for dense models; for MoE, EP usually spans the DP×TP group.

A worked example, Llama 3.1 405B in FP8 on H100 nodes (8 × 80 GB):

• Weights are ~405 GB. One node has 640 GB, so TP=8, PP=1 fits on a single node with room for KV cache.
• In BF16 (~810 GB) it no longer fits, so go TP=8, PP=2 across two nodes.
• Need 10x the throughput? DP=10 replicas of that unit, behind a router.

Three layers of software make this run in production:

vLLM is the engine. PagedAttention manages KV cache like virtual memory pages, so many requests share GPU memory without fragmentation. Continuous batching adds and removes requests every step. It supports TP, PP, DP and EP, chunked prefill, prefix caching, and uses Ray or native multiprocessing to coordinate multi-node workers. SGLang and TensorRT-LLM fill the same slot.

NVIDIA Triton Inference Server is the enterprise serving shell. It hosts backends (TensorRT-LLM, vLLM, Python, ONNX) behind standard HTTP/gRPC endpoints, with dynamic batching, model ensembles, and Prometheus metrics. NVIDIA Dynamo is its newer, LLM-native successor for disaggregated, multi-node serving.

Ray Serve is the orchestration layer in Python. Each model replica is a Ray actor that can claim a placement group of GPUs across nodes. Ray Serve handles request routing, batching, autoscaling and multi-model composition, and Ray Serve LLM wraps vLLM with these primitives out of the box.

from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app

config = LLMConfig(
    model_loading_config={"model_id": "llama-405b",
                          "model_source": "meta-llama/Llama-3.1-405B-Instruct-FP8"},
    deployment_config={"autoscaling_config": {"min_replicas": 2, "max_replicas": 10}},
    engine_kwargs={"tensor_parallel_size": 8, "pipeline_parallel_size": 1},
)
serve.run(build_openai_app({"llm_configs": [config]}))

Engine choices decide how one model instance spans GPUs. The next question is how to schedule hundreds of those instances on a fleet. That's Kubernetes.

7. Kubernetes: running it all in production

Kubernetes was built for stateless, single-pod services. A TP=8, PP=2 model replica is the opposite: 16 GPUs across two nodes that must start together, die together, and sit on the same high-speed fabric. The ecosystem has grown a stack of components to close that gap.

1. Exposing GPUs and fabric. The NVIDIA GPU Operator installs drivers, the device plugin, and DCGM metrics, so pods can request nvidia.com/gpu: 8. Dynamic Resource Allocation (DRA) goes further, letting pods claim specific GPUs, MIG slices, or NVLink-connected sets. The NVIDIA Network Operator (SR-IOV, RDMA, Multus) gives pods direct InfiniBand/RoCE access for PP, EP and KV transfers.

2. Multi-node replicas as one unit. LeaderWorkerSet (LWS) is the Kubernetes SIG API for this: one leader pod plus N worker pods form a group, scaled and restarted as one replica. If any worker dies, the whole group restarts, which matches how NCCL groups fail.

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: llama-405b
spec:
  replicas: 4                  # DP = 4
  leaderWorkerTemplate:
    size: 2                    # PP = 2 nodes per replica
    restartPolicy: RecreateGroupOnPodRestart
    leaderTemplate:
      spec:
        containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          command: ["sh", "-c"]
          args:
          - ray start --head --port=6379 &&
            vllm serve meta-llama/Llama-3.1-405B-Instruct
              --tensor-parallel-size 8 --pipeline-parallel-size 2
          resources:
            limits: { nvidia.com/gpu: 8 }   # TP = 8 per node
    workerTemplate:
      spec:
        containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          command: ["sh", "-c"]
          args: ["ray start --address=$(LWS_LEADER_ADDRESS):6379 --block"]
          resources:
            limits: { nvidia.com/gpu: 8 }

3. Gang and topology-aware scheduling. Half-scheduled replicas waste GPUs forever. Kueue and Volcano provide gang scheduling (all pods or none) and quota management. Topology-aware placement keeps a replica's pods on the same rack or NVLink domain.

4. LLM-aware routing. Round-robin load balancing is wrong for LLMs: requests differ wildly in cost, and the best replica is the one that already has your prompt prefix in its KV cache. The Kubernetes Gateway API Inference Extension adds an InferencePool resource and an endpoint picker that routes on queue depth, KV-cache utilization, prefix-cache hits and LoRA adapters.

5. Opinionated full stacks. You rarely assemble all of this by hand:

Project	What it gives you
vLLM production-stack	Helm chart: vLLM replicas, prefix-aware router, LMCache, Grafana dashboards
llm-d	Kubernetes-native distributed vLLM: PD disaggregation, wide EP, cache-aware routing via Inference Gateway
NVIDIA Dynamo + Grove	Disaggregated serving across vLLM/SGLang/TRT-LLM; Grove handles multi-role gang scheduling
KServe	InferenceService CRD with a vLLM-based LLM runtime, scale-to-zero, canary rollouts
KubeRay	RayService CRD that runs Ray Serve LLM apps with zero-downtime upgrades

6. Autoscaling on the right signal. CPU utilization is meaningless for GPU serving. Scale on queue depth, KV-cache usage, or TTFT/TPOT SLOs via KEDA or the HPA with custom metrics. With PD disaggregation, scale prefill and decode pools independently.

7. Cold starts. A 400 GB model takes minutes to pull. Teams pre-stage weights on node-local NVMe or a shared PVC, use streaming loaders (e.g., the Run:ai Model Streamer), and keep a warm floor of replicas.

Cheat sheet

Strategy	Splits	Main comms	Best link	Solves
DP	Requests	None (inference)	Any	Throughput
FSDP / ZeRO-3	Parameter storage	All-gather per layer	NVLink / IB	Memory (mostly training, offline inference)
PP	Layers	Point-to-point activations	InfiniBand	Capacity across nodes
TP	Matrices	2 all-reduces per layer	NVLink	Latency + single-node capacity
EP	MoE experts	2 All-to-Alls per MoE layer	NVLink + RDMA	MoE capacity + throughput
PD disaggregation	Phases	KV-cache transfer	RDMA	TTFT/TPOT interference

A practical order of operations: fit the model with TP inside a node; if it still doesn't fit, add PP (or EP for MoE) across nodes; scale throughput with DP; once SLOs bite, disaggregate prefill and decode. Then let Kubernetes treat each multi-GPU replica as one schedulable unit, routed by an LLM-aware gateway.

Scaling LLM inference isn't one trick. It's matching each kind of split to the right layer of your network, and then teaching your orchestrator that a "pod" might be sixteen GPUs.

References

• PyTorch DistributedDataParallel
• PyTorch FullyShardedDataParallel
• DeepSpeed ZeRO
• PyTorch torch.distributed.pipelining
• DeepSpeed Pipeline Parallelism
• PyTorch torch.distributed.tensor.parallel
• NVIDIA Megatron-LM
• NVIDIA TensorRT-LLM overview
• DeepSpeed Mixture-of-Experts
• vLLM documentation
• vLLM production-stack: disaggregated prefill
• SGLang
• NVIDIA Triton Inference Server
• Ray Serve
• Kubernetes LeaderWorkerSet