CI: HyperPod integration tests fail due to missing hyperpod CLI and missing cluster 'riv-rig'

[Mattral]

PySDK Version

• PySDK V2 (2.x)
• PySDK V3 (3.x)

Describe the bug
Several GPU integration tests fail in CI because the HyperPod CLI Python package/CLI is not installed and the SageMaker HyperPod cluster named "riv-rig" is not present in the test environment. Failures observed in the run https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506 include:

• ModuleNotFoundError: No module named 'hyperpod_cli'
• FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod' (the CLI executable)
• RuntimeError / botocore error: Failed to describe cluster 'riv-rig': Cluster with name riv-rig not found

Relevant failing tests:

• tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod
• tests/integ/train/test_cpt_data_mixing_hyperpod.py::test_cpt_trainer_nova_micro_with_data_mixing_hyperpod
• tests/integ/train/test_sft_data_mixing_hyperpod.py::test_sft_trainer_nova_micro_data_mixing_hyperpod
• tests/integ/train/test_nova_hyperpod.py::test_sft_trainer_nova_micro_hyperpod_lora

Stack / file references:

• Import failure originates in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py where the code tries "import hyperpod_cli" (see build_hyperpod_datamix_recipe_from_context).
• DescribeCluster failure occurs when code calls sagemaker_client.describe_cluster(ClusterName='riv-rig').

To reproduce

1. Check out commit (used in the failing run): 6659b11
2. Run the GPU integ tests (same CI job): e.g., run the workflow or locally run:
   • pip install -r requirements-dev.txt
   • pytest tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod -q
     or run the entire integration test set used by the GPU job.
3. Observe ModuleNotFoundError or describe_cluster ResourceNotFound in the logs.

Expected behavior
Integration tests should not fail due to missing test dependencies or absent external infra. Possible acceptable behaviors:

• CI installs the hyperpod package/CLI before running tests, and either a test cluster exists or tests stub/skip interactions with cluster resources; or
• Tests that depend on the hyperpod CLI/cluster are mocked or skipped when running in CI without the required infra.

Screenshots or logs
Key excerpts:

• Module error: "ModuleNotFoundError: No module named 'hyperpod_cli'"
• Exec error: "FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod'"
• Cluster error: "RuntimeError: Failed to describe cluster 'riv-rig': An error occurred (ResourceNotFound) when calling the DescribeCluster operation: Cluster with name riv-rig not found"
  Full job: https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506

System information

• SageMaker Python SDK version: repo commit 6659b11 (CI run)
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): N/A (integration tests for HyperPod / Nova fine-tuning)
• Framework version: N/A
• Python version: Python 3.10.13 (from CI logs)
• CPU or GPU: GPU integration tests (workflow is "GPU Integ Tests")
• Custom Docker image (Y/N): CI uses CodeBuild / runner image (see workflow)

Additional context

• The test code expects the hyperpod CLI Python package (hyperpod_cli) importable and the hyperpod executable on PATH. The code that imports hyperpod_cli is in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py (build_hyperpod_datamix_recipe_from_context).
• The tests also rely on an existing HyperPod cluster named 'riv-rig' being reachable via SageMaker API. CI account does not have that cluster leading to ResourceNotFound.
• The logs also show a pip dependency warning: virtualenv requires platformdirs<4 but platformdirs 4.10.1 was installed — this is not causing the test failures but may indicate CI environment drift.

Recommended remediation

• For CI: add installation of the hyperpod package/CLI to the test setup (pip install hyperpod) or vendor a minimal fake hyperpod module for tests that only need to write recipe files.
• Make tests resilient to missing cluster:
  • Use botocore.stub.Stubber or a pytest fixture to stub describe_cluster responses in CI.
  • Or skip HyperPod integration tests when a real cluster isn't available (pytest.skip with a clear message).
• Prefer mocking the external infra in CI so tests are deterministic, or make infra provisioning explicit in CI docs if real cluster is required.
• Optionally pin platformdirs in test environment to avoid pip conflict warnings.