AI 基礎設施
AWS ParallelCluster 3.16 Adds Node Diagnostics, but the Upgrade Also Changes IAM, NFS, and the Container Stack
The new AMI includes `pcluster-diag`, which can run role-specific checks of Slurm, IMDS, directory services, and Lustre status and produce structured reports. The upgrade also ends support for Amazon Linux 2 and AWS Batch, while introducing permission, network egress, and runtime changes that require advance preparation.
AWS announced the general availability of ParallelCluster 3.16 on August 24. Its headline addition is `pcluster-diag`, which is installed with the official AMI. The tool reads the node role and cluster configuration, then selectively runs nine categories of checks, including IMDS, reserved accounts and file permissions, ParallelCluster management processes, the `clustermgtd` heartbeat, directory services, Slurm accounting, and FSx for Lustre. Results are written as JSON. A `FAILURE` or `CHECK_ERROR` causes the command to fail, while a `WARNING` still returns success, so operations automation integrations cannot rely on the exit code alone.
An [independent hands-on test](https://dev.classmethod.jp/articles/aws-parallelcluster-3-16-0-released-pcluster-diag/) completed nine checks on an Ubuntu 24.04 node in about four seconds and successfully triggered a failure by setting incorrect permissions on `munge.key`. However, the tool must run as root, and its report includes the complete `cluster_config` and `dna_json`. Before uploading reports to a centralized system, organizations should assess whether the topology, resource names, and configuration details they contain constitute sensitive data.
The [3.16.0 release notes](https://github.com/aws/aws-parallelcluster/releases/tag/v3.16.0) include several other changes that could affect AI/HPC clusters. During cluster creation, EBS mounts affected by transient IMDS issues are now retried; login node updates are coordinated centrally by the head node; and bootstrap temporary files have been moved out of `/tmp`, improving compatibility with custom AMIs that mount it with `noexec`. On the other hand, the CLI now requires the `tag:GetResources` permission; managed NFS servers are required to use NFSv4; and image builds in isolated subnets must allow access to `amazon-efs-utils.aws.com`. Enroot has been upgraded from 3.x to 4.2.1, while Pyxis, CUDA, the NVIDIA driver, EFA, and Slurm have also been updated. Existing container workloads should undergo compatibility testing first. Support for Amazon Linux 2 and the AWS Batch scheduler has also ended, so this update should not be treated as a risk-free addition of diagnostic functionality.