Run multi-node inference workloads using Ray LLM Deep Learning Containers

This post shows how to deploy one large language model (LLM) across two GPU nodes on Amazon Elastic Kubernetes Service (Amazon EKS) with the multi-node Ray LLM Deep Learning Containers . It uses separate prefill and decode nodes behind one OpenAI-compatible endpoint. This kind of multi-node setup is common for tensor, pipeline, and data parallelism, as well as prefill/decode disaggregation inference workloads. In a previous post , we introduced the Ray Serve DLC and used it to stand up a small model behind an HTTP endpoint on a single GPU node. You had a working inference endpoint, and the container keeps the GPU stack compatible, including CUDA libraries, so you did not need to write boilerplate code. Ray is a distributed compute framework, so Ray Serve deployments can span multiple nodes. Customers use multi-node Ray Serve for several reasons. Some want to run foundation or open-weight models that are too large for a single GPU. Others need more throughput than one replica can deliver. Still others want to separate the two phases of inference to improve performance. In each case, customers must build the dependency chain themselves: choosing compatible libraries for the GPU stack and debugging issues when a component drifts out of alignment. This is undifferentiated work that takes time and effort away from delivering models. AWS Deep Learning Containers (DLCs) have long solved this kind of problem for training and inferencing workloads. DLCs are pre-built, performance-optimized Docker images that bundle a framework, its dependencies, and the GPU stack into a tested, patched combination you can pull and use immediately. With the launch of the Ray LLM DLC for multi-node deployments, that same approach now extends to multi-node inference. You get a container purpose-built for serving models behind an HTTP endpoint, maintained and tested by AWS, with the full inference stack already assembled. This post walks through implementing multi-node disaggregated inference using multi-node Ray LLM DLC: deploying a single large language model across two GPU-powered nodes on Amazon EKS cooperating behind a single endpoint. While one of the GPU nodes handles the prefill phase, the other GPU node handles the decode phase. Separating the phases prevents compute-heavy long prompts from stalling token generation for requests already in progress. This approach is well suited for workloads such as Retrieval Augmented Generation (RAG), document summarization, and agentic applications. We use the multi-node Ray LLM DLC, which ships Ray Serve LLM , NVIDIA Inference Transfer Library (NIXL) , and the Elastic Fabric Adapter (EFA) stack, and exposes an OpenAI-compatible API with no application code. The complete code is available in the accompanying repository . Why prefill/decode (PD) disaggregation LLM inference has two distinct phases. Prefill processes the entire input prompt in parallel and builds the key-value (KV) cache. Decode then generates the response one token at a time, reading and extending that cache. Prefill is compute bound, and Decode is bound by memory bandwidth. When both phases share one GPU, they contend for the same resources, and the bursty, compute-heavy prefill work interferes with the steady, latency-sensitive decode work. Separating the prefill phase from the decode phase means you can select the hardware and serving properties for each phase, improving overall performance at scale. PD disaggregation places the two phases on separate nodes, so each phase runs without interfering with the other and can be scaled independently. The trade-off is that the KV cache built during prefill must be handed to the decode node, and that transfer should be fast enough not to erase the benefit. That is exactly why the interconnect between the nodes matters. What the Ray LLM DLC provides AWS DLCs are pre-built, performance-optimized Docker images that bundle a framework, its dependencies, and the GPU stack into a tested, patched combination you can pull and use immediately. All components are validated together before each release, so there is no version drift between the CUDA runtime, the framework, and the serving layer. Security patches are applied at build time. The Ray LLM DLC extends this to distributed LLM serving. It ships vLLM as the inference engine, Ray Serve’s HTTP as the serving layer, NIXL for KV-cache transfer between nodes, and the EFA stack ( libfabric and aws-ofi-nccl ) for high-bandwidth node-to-node communication. Because the Ray Serve LLM library includes a built-in application builder, the entire serving application is a block of YAML. There’s no handler class, no model archiver, and no custom image to build. A single helper, build_pd_openai_app , takes a prefill configuration and a decode configuration and wires the KV-cache transfer between them behind one OpenAI-compatible endpoint. Architecture The following diagram shows how the deployment is laid out on Amazon EKS. A user request reaches the Ray head on the CPU system node, and the two GPU worker nodes run the prefill and decode phases of the model. Figure 1: Multi-node inference architecture on Amazon EKS, with the Ray head on a CPU system node and the prefill and decode phases on two GPU worker nodes The deployment is an EKS cluster inside a virtual private cloud (VPC). Nodes run in private subnets with outbound access through a NAT Gateway. A CPU system node (m7i.xlarge) hosts system workloads, the KubeRay operator, and the Ray head pod, which is CPU-only. Two GPU worker nodes (g6.8xlarge) run the Ray workers: one serving the prefill phase, and the other serving the decode phase. Both nodes pull the Ray LLM DLC from Amazon Elastic Container Registry (Amazon ECR) . KubeRay watches the RayService you apply and wires the head-to-worker join automatically. Ray Serve places the prefill and decode deployments on the two GPU workers. The KV cache flows from the prefill worker to the decode worker over NIXL, using EFA through NIXL’s libfabric backend. The GPU node group is pinned to a single Availability Zone (AZ) because EFA cannot cross AZs. Prerequisites To follow along with this post, you need: An AWS account with billing enabled. Sufficient service quota for running two g6.8xlarge instances (at least 64 vCPU) in your target Region. AWS Command Line Interface (AWS CLI) with credentials configured. kubectl CLI for communicating with a Kubernetes cluster’s control plane. eksctl CLI for creating and managing clusters on EKS. helm (package manager for Kubernetes) to install KubeRay operator. envsubst utility (part of the GNU gettext) to render the manifest. Verify that your AWS credentials are active with the following command: aws sts get-caller-identity Use the pre-built application to serve inference The application is the serveConfigV2 block in manifest/rayservice.yaml . It uses build_pd_openai_app , which takes a prefill_config and a decode_config and connects the KV-cache transfer between them: applications: - name: qwen import_path: ray.serve.llm:build_pd_openai_app # OpenAI-compatible PD app, built in route_prefix: / args: prefill_config: model_loading_config: { model_id: qwen3.5-9b, model_source: Qwen/Qwen3.5-9B } engine_kwargs: max_model_len: 4096 dtype: bfloat16 gpu_memory_utilization: 0.90 enforce_eager: true # Skip CUDA graph capture to reduce peak GPU memory kv_transfer_config: kv_connector: NixlConnector kv_role: kv_both engine_id: prefill kv_connector_extra_config: { backends: ["LIBFABRIC"] } deployment_config: { num_replicas: 1 } decode_config: model_loading_config: { model_id: qwen3.5-9b, model_source: Qwen/Qwen3.5-9B } engine_kwargs: max_model_len: 4096 dtype: bfloat16 gpu_memory_utilization: 0.90 enforce_eager: true kv_transfer_config: kv_connector: NixlConnector kv_role: kv_both engine_id: decode kv_connector_extra_config: { backends: ["LIBFABRIC"] } deployment_config: { num_replicas: 1 } A few settings carry most of the weight. max_model_len: 4096 sizes the context to