repository·master·Indexed 43 days ago
https://github.com/imbue-ai/cluster-healthA collection of specialized scripts for managing, diagnosing, and maintaining large-scale H100 GPU clusters. Includes tools for GPU stress testing, host health validation, NVLink and InfiniBand communication testing, IB burn-in workloads, and network port management via UFM event logs.
host_validation module contains tests to verify inter-GPU communication. This includes checking local GPU-to-GPU communication via NVLink and remote GPU-to-GPU communication across machines via InfiniBand.ib_burn script generates a comprehensive burn-in workload specifically designed for InfiniBand (IB) fabrics. The goal is to exercise every available link in the fabric to identify potential failures.ufm_events script parses UFM (Unified Fabric Manager) event logs and other system logs to identify relevant network events. It is used to determine which specific network ports should be disabled to maintain cluster stability.health_checks module contains various diagnostic checks to determine host health and provides automated solutions for common hardware or software issues encountered in the cluster.gpu_stress_test module provides checks to ensure that GPUs on a machine can successfully allocate large tensors and perform standard operations without failure.