AWS Brings Ray Into SageMaker HyperPod With Recovery Tools and Tiered Cache on EKS

AWS has integrated Ray into SageMaker HyperPod on Amazon EKS, putting cluster management, job submission, dashboards, and hung-job-detection configuration into SageMaker Studio. HyperPod supplies the recovery and storage features underneath, while the integration remains compatible with KubeRay and standard Ray APIs.

Ray is an open-source framework for scaling distributed Python workloads across GPU clusters, including training with Ray Train and serving with Ray Serve. On Kubernetes, KubeRay manages RayCluster, RayJob, and RayService resources. AWS says users previously had to write YAML manifests, rebuild Docker images for dependency changes, use kubectl port-forwarding for the Ray Dashboard, and configure Prometheus and Grafana themselves.

Studio can create and manage Ray clusters, open Ray and Grafana dashboards, attach JupyterLab or Code Editor workspaces, submit distributed jobs, and configure hung-job detection. Dashboard access uses a short-lived IAM-authenticated URL scoped to the cluster creator; remote endpoints also support job submission and log retrieval without a local port-forward.

For remote workloads, AWS provides the toolkit-for-ray-on-sagemaker-ai Python package, which handles SageMaker-aware addresses and EKS API credentials through IAM authentication. It lets Studio, laptops, and CI/CD pipelines use standard Ray job-submission APIs. An attached JupyterLab or Code Editor workspace joins the cluster as a zero-compute worker node, giving the notebook native Ray driver access.

When HyperPod replaces a faulty node, Ray reschedules worker pods to the healthy node. For stalled Ray Train workloads, a per-node monitoring agent watches node- and job-level signals and notifies users through CloudWatch and the Ray Train Grafana dashboard. If configured to cancel, HyperPod terminates the stalled worker process and Ray Train can restart workers from the last checkpoint through FailureConfig.

The amzn-sagemaker-checkpointing library writes checkpoints to local disk, then uploads them asynchronously to Amazon S3. On restart, it checks HyperPod Tiered Storage first, which AWS says can shorten large-model recovery compared with restoring directly from S3.

For inference, a JumpStart loader can place catalog model weights directly into Ray Serve endpoints. Those deployments can use Managed Tiered KV Cache, which stores attention key-value vectors in node CPU memory and HyperPod Tiered Storage. AWS says the setup reduces time to first token for multi-turn and long-document workloads with minimal code changes, but did not publish performance figures.

AWS says existing Ray scripts and workflows can run without modification. The service is not standalone, however: users need a HyperPod cluster with EKS orchestration, a SageMaker Studio domain, and installed Spaces, observability, KubeRay, and Ray Endpoint Operator components. Availability is limited to AWS Regions that support HyperPod EKS.