Share GPU clusters across teams with isolation and fairness using Amazon SageMaker HyperPod

Multiple teams within the same company increasingly need shared access to expensive GPU clusters for their generative AI operations, while maintaining isolation boundaries, resource fairness, and operational independence. Consider a data science team training large language models, a computer vision group running inference workloads, and a research team experimenting with new model architectures. All of them might need access to the same cluster. Without a well-designed multi-tenant (multi-team) architecture, organizations face uncontrolled resource consumption, weak isolation between teams, an inability to attribute shared GPU costs to the teams that incur them, and administrative overhead that slows down innovation. Amazon SageMaker HyperPod is a purpose-built AI service that simplifies the management of large-scale compute clusters for gen AI workloads. It provides resilient, optimized clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS) or Slurm, so organizations can run distributed training, interactive development, and model inference at scale. At the same time, it automatically handles node health monitoring, fault recovery, and cluster lifecycle management. In this post, we present a reference architecture for building a multi-tenant environment on Amazon SageMaker HyperPod with EKS. This architecture uses AWS IAM Identity Center for centralized authentication, per-team SageMaker AI domains for a tailored user experience, Kubernetes namespaces for workload isolation, HyperPod Task Governance for fair resource allocation, and namespace-level cost allocation for per-team spend visibility and chargeback. By the end of this post, you will have a clear blueprint for multiple teams to efficiently share a single HyperPod EKS cluster. Architecture overview The following diagram illustrates the high-level architecture of a multi-tenant HyperPod EKS deployment. In this example, two teams (Team A and Team B) share a single HyperPod EKS cluster, each operating within their own isolated namespace. Figure 1: High-level multi-tenant architecture for two teams sharing one HyperPod EKS cluster The architecture is structured as a layered flow from left to right, connecting user identity through authorization controls and into isolated workload namespaces on the cluster. Users and authentication On the far left, individual users from each team (User 1 from Team A, User 2 from Team B) interact with the system through two paths. Both paths authenticate through the AWS IAM Identity Center Portal, which federates with an external identity provider (such as Microsoft Entra ID) shown at the bottom-left. The first path is through CLI access. Users authenticate with aws sso login , which redirects them to the Identity Center portal, and then obtain temporary credentials from their team’s permission set to submit tasks directly to the EKS cluster with kubectl . In the diagram, the pink arrow flows from the CLI terminal across the top directly into the HyperPod EKS cluster. The second path is through the Identity Center portal directly, where users select the SageMaker Studio application to sign in to their team-specific SageMaker AI domain. Each team has a corresponding permission set (TeamA permission set, TeamB permission set) that carries the AWS Identity and Access Management (IAM) policies needed for CLI workflows. Identity Center automatically provisions an IAM role for each permission set, shown in the diagram as TeamA-permissionset-role and TeamB-permissionset-role (labeled “CLI/Console role”). This role serves as the IAM principal when users authenticate through aws sso login . SageMaker AI domains From the Identity Center portal, users are routed to their team-specific SageMaker AI domain. Each domain (SageMaker AI domain Team A and SageMaker AI domain Team B) provides a dedicated Amazon SageMaker Studio GUI and is configured with a team-specific execution role ( TeamA-role and TeamB-role respectively). These domains serve as the primary workspace interface, so users can submit tasks from the GUI (shown by the pink arrows flowing toward EKS). EKS access control At the EKS boundary, access entries map IAM roles to Kubernetes permissions. The diagram shows access entries for TeamA-role and TeamB-role (the Studio execution roles), which authorize requests originating from the SageMaker Studio GUI. Access entries must also be configured for the Identity Center-provisioned CLI/Console roles ( TeamA-permissionset-role and TeamB-permissionset-role ) to authorize requests arriving through kubectl . All access entries are associated with managed or custom role-based access control (RBAC) policies (represented by the key icon) and scoped to the team’s designated namespace. As a result, whether access originates from Studio or the CLI, users can only interact with resources in their own namespace. HyperPod EKS cluster The cluster itself is depicted with two cross-cutting platform layers at the top: HyperPod Observability (for monitoring and dashboards) and HyperPod Task Governance (for compute quota management and scheduling priorities). Below these layers, the cluster is partitioned into Namespace A (Team A) and Namespace B (Team B). Within each namespace, teams can run their own HyperPod Spaces (interactive development environments), HyperPod PyTorch jobs (distributed training workloads), and HyperPod Inference endpoints (model serving). Storage Beneath the cluster, the architecture includes two storage tiers. The first is a POSIX-compliant file system (Amazon FSx for Lustre or Amazon FSx for OpenZFS) organized into per-team shared directories ( /fsx/TeamA , /fsx/TeamB ) and per-user home directories ( /home/User1 , /home/User2 ). The second is per-team or shared Amazon Simple Storage Service (Amazon S3) buckets for object storage, governed by the team’s IAM execution role. This architecture isolates each team from authentication through authorization to workload execution, while sharing expensive GPU infrastructure efficiently. Authentication and access control The foundation of any multi-tenant system is robust authentication: verifying who users are before they interact with any resource. In this architecture, AWS IAM Identity Center serves as the centralized authentication layer, federating with an external identity provider to manage user identities and group memberships. Why AWS IAM Identity Center AWS IAM Identity Center (successor to AWS Single Sign-On) provides a single place to manage workforce identities across AWS accounts and applications. For a multi-tenant HyperPod deployment, it offers several key capabilities: Centralized identity management – Rather than maintaining separate user databases per AWS service, Identity Center provides a single source of truth for all user identities and their group memberships. Federation with existing identity providers – Most enterprises already manage their workforce identities in systems like Microsoft Entra ID (formerly Azure AD), Okta, or Ping Identity. Identity Center integrates with these providers, so organizations can reuse their existing identity infrastructure without duplicating user accounts. Native integration with SageMaker AI – SageMaker AI domains support Identity Center authentication, so users can sign in to SageMaker Studio through their corporate identity provider with single sign-on (SSO). AWS account access – Identity Center can also grant users access to the underlying AWS account with specific permission sets, which support CLI workflows alongside the Studio GUI experience. Required for Amazon Managed Grafana – Amazon Managed Grafana uses Identity Center as its authentication mechanism for workforce users, making it the natural choice when teams also need access to observability dashboards for monitoring their workloads. Learn more: What is IAM Identity Center Configuring Identity Center with an external identity provider In this reference architecture, we use Microsoft Entra