As HPC centres rapidly acquire GPU clusters for large-scale AI training, the community lacks a standardised methodology for assessing whether a cluster is operationally healthy and ready for production workloads.
Existing benchmarks measure peak capability LINPACK reports FLOPS, HPL-MxP tests mixed-precision arithmetic, and NCCL-tests measure collective bandwidth but none captures the operational health dimensions that determine whether a distributed training run will actually complete successfully.
Thermal headroom under sustained load, InfiniBand fabric consistency across all rails, NVLink and NVSwitch error rates under realistic collective traffic patterns, and storage I/O stability during checkpoint writes all remain untested by current acceptance procedures.
Published data from hyperscaler operations (Meta HPCA 2025, ByteDance SMon) confirms that the dominant causes of training disruption are not peak performance shortfalls but rather subtle, sustained degradation in these operational dimensions degradation that standard benchmarks are structurally blind to.
This talk proposes an open framework for GPU cluster health benchmarking organised around four dimensions:
(1) Compute health : sustained SM utilisation under realistic workloads, memory bandwidth consistency, and ECC error rate trends (noting that the widely-used DCGM_FI_DEV_GPU_UTIL metric measures binary kernel activity rather than true SM occupancy, systematically overstating actual utilisation); (2) Network health : all-reduce latency variance across topology positions, adaptive routing effectiveness, and congestion response under multi-tenant traffic; (3) Thermal resilience : time-to-throttle under sustained load, cooling asymmetry across rack positions, and ambient temperature sensitivity curves; and (4) Storage readiness: checkpoint write throughput, metadata operation latency, and parallel I/O scaling behaviour.
We present initial results from applying this framework to production clusters, identifying common failure patterns that escape traditional acceptance testing but manifest during sustained training runs.
These include thermal-induced straggler effects invisible to per-node metrics, InfiniBand rail imbalances that only surface under all-to-all traffic, and storage contention during synchronous checkpointing at scale.
We propose this framework as a community standard an open health certification methodology that complements existing performance benchmarks with operational readiness metrics and invite collaboration from HPC centres, AI ML teams, and the broader community to refine and extend it.
