DOCSEARCHbyAlgolia
Submit Docs

NVSentinel

repository·main·Indexed 43 days ago

https://github.com/nvidia/nvsentinel

A Kubernetes-native tool designed to detect, protect, and remediate GPU faults. NVSentinel automates the identification of hardware issues via DCGM and system logs, cordons and drains affected nodes to protect running jobs, and performs automated repairs such as GPU resets or reboots. It features an event-driven architecture using CEL (Common Expression Language) rules for fault quarantine and supports extensibility through custom remediation workflows and custom drain mechanisms via Kubernetes Custom Resources (CRs).

Tokens
461.8K
Snippets
862
Records
1.3K
Agent score
56%

What's inside NVSentinel

  1. Overview of NIC Link State Detection

    main

    The NIC Health Monitor's Link State Detection component is designed to detect hard failures in network interfaces to prevent 'Grey Failures' and 'straggler effects' in GPU clusters. It focuses on detecting when a single degraded link might throttle performance across the cluster.

    Scope of Detection:

    • Hard UP/DOWN transitions: Complete loss of connectivity.
    • Device disappearance: The NIC is no longer visible in sysfs (e.g., fell off the PCIe bus).
    • Physical state changes: Ports being disabled, in polling mode, or in error recovery.
    • Uncabled port anomaly detection: Identifying cards that have fewer active ports than their peers through homogeneity checks.
    • Management NIC auto-exclusion: Automatically excluding NICs located on NUMA nodes that do not contain a compute GPU.

  2. Overview of NIC Health Monitor Design

    main

    The NIC Health Monitor is a monitoring solution designed to detect and report network interface failures in GPU clusters. It specifically targets Grey Failures—subtle network degradations that can throttle distributed training workloads across thousands of GPUs.

    The monitoring capability is divided into three specialized detection areas:

    1. Link State Detection: Monitors UP/DOWN status and device disappearance. It handles binary state changes, classifies NIC roles (compute, storage, or management), and identifies uncabled port anomalies.
    2. Link Counter Detection: Monitors error rates and threshold violations. This includes Bit Error Rate (BER) tracking, FEC (Forward Error Correction) exhaustion prediction, and congestion detection.
    3. Syslog Detection & Correlation: Monitors kernel logs for driver or firmware errors and identifies correlated failure patterns.

  3. Overview of Kubernetes Object Monitor

    main

    The Kubernetes Object Monitor is a policy-based watchdog that watches any Kubernetes resource (Nodes, Pods, Custom Resources, etc.) and generates health events when they enter unhealthy states.

    It uses CEL (Common Expression Language) to define what constitutes an "unhealthy" state. When a CEL expression evaluates to true, an unhealthy event is generated; when it evaluates to false, a healthy event is automatically sent. This allows you to integrate existing Kubernetes-based monitoring into NVSentinel without writing custom code.

  4. What is NVSentinel?

    main
    NVSentinel is an automated GPU fault detection and remediation system designed for Kubernetes. It monitors GPU health, system logs, and cloud provider maintenance events to automatically perform actions such as cordoning faulty nodes, draining workloads, and triggering break-fix workflows (like GPU resets or node reboots).

  5. Key capabilities of the NIC Health Monitor

    main

    The NIC Health Monitor provides several advanced features for managing network health:

    • Deterministic failure thresholds: Uses IBTA specifications, cloud provider heuristics (Azure, AWS), and vendor documentation.
    • Allowlisted counter configuration: Operators can select specific NIC counters, set custom delta or velocity-based thresholds, and define fatal/non-fatal severity per counter.
    • Pre-failure prediction: Detects Bit Error Rate (BER) climbing before FEC exhaustion (e.g., using the IBTA 10E-12 BER threshold of 120 errors/hour).
    • Persistent local state: Uses snapshots for ports/devices, counter timestamps, and breach flags to survive pod restarts. Outstanding FATAL latches remain active until matching recovery evidence is received.
    • Zero-configuration NIC role classification: Automatically classifies NIC roles using NUMA locality and an nvidia-smi topo -m matrix (consumed from gpu_metadata.json). This supports x86 DGX/HGX, Grace-based superchips (GB200, GH200), and various cloud platforms.

  6. Navigate NVSentinel Documentation

    main

    NVSentinel documentation is organized by user role and technical depth. Use the following guide to find relevant information:

    • New Users: Start with OVERVIEW.md for a high-level introduction to the system.
    • Platform Engineers: Refer to INTEGRATIONS.md to learn about integrating NVSentinel with scheduling, monitoring, and remediation systems (e.g., taints, node conditions, and drain behavior).
    • Architects & Developers: Use DATA_FLOW.md to understand the end-to-end data lifecycle from failure detection to remediation.
    • SREs & Monitoring Teams: Consult METRICS.md for the complete catalog of Prometheus metrics for dashboarding and alerting.
    • Developers (Extending the system): Use the tutorials/ directory for step-by-step guides on writing new health monitors, preflight checks, drain plugins, or custom remediation actions.
    • Operators: Use runbooks/ for troubleshooting and configuration/ for component-specific setup instructions.

  7. PostgreSQL Support in NVSentinel store-client SDK

    main

    The store-client SDK supports PostgreSQL through a multi-layered, database-agnostic architecture. This allows NVSentinel services (like node-drainer or fault-quarantine) to interact with either MongoDB or PostgreSQL using the same high-level interfaces.

    Key capabilities include:

    • Database-agnostic queries: Use a single query builder to target either database.
    • Native SQL performance: PostgreSQL implementation utilizes native SQL queries with JSONB operators.
    • Provider abstraction: Switch between MongoDB and PostgreSQL via environment variables without changing application code.
    • Change stream support: PostgreSQL uses a polling-based mechanism for change detection.

  8. Supported NIC Hardware

    main

    The current implementation of the NIC Health Monitor is scoped to Mellanox/NVIDIA InfiniBand and RoCE devices.

    • Detection Method: Identifies devices via name matching mlx5_* or driver symlinks.
    • Capabilities: Performs counter monitoring (fatal counters), state monitoring, and kernel log analysis.

    Future Support Roadmap:

    • AWS EFA (device names matching rdmap\d+s\d+)
    • Plain Ethernet (operstate = down via /sys/class/net/{interface}/operstate)
    • TCPXO (TCP Express Offload)

  9. Configure the In-Cluster File Server

    main

    The In-Cluster File Server is an nginx-based server that stores diagnostic log bundles collected by the fault-remediation.logCollector. It organizes logs on a persistent volume by node name and timestamp, serving them over HTTP within the cluster.

    Note: This module is only required when fault-remediation.logCollector.enabled is set to true.

  10. What is a health monitor in NVSentinel

    main

    A health monitor is a standalone process responsible for detecting faults and reporting them to NVSentinel. It follows a simple lifecycle:

    1. Observe: Monitor a source (e.g., GPU via DCGM, kernel logs, cloud APIs, or /proc files).
    2. Decide: Determine if a node or component is healthy or unhealthy.
    3. Report: Send a HealthEvent to the platform-connector via a local gRPC Unix domain socket.

    Key Architectural Constraints:

    • No Kubernetes API access required: Monitors do not perform cordoning or draining; they only emit events. The platform-connector handles the Kubernetes interaction.
    • Transport: Communication happens over a Unix socket on the node, typically shared via a hostPath volume.
    • Contract: The interface is limited to a single gRPC method and a single message type.
    graph LR
    YM["Your Monitor<br/>(detect + report)"] -->|"gRPC over<br/>unix:///var/run/nvsentinel.sock"| PC["Platform Connector"]
    PC -->|"node condition<br/>(fatal events)"| K8S["Kubernetes"]

  11. Overview of NIC Syslog Detection

    main

    The Syslog Health Monitor is a component of NVSentinel designed to detect NIC failures that standard hardware polling (like state checks or counter reads) might miss. Specifically, it monitors the driver/firmware interface to identify issues where the NIC appears healthy (e.g., Link is UP) but is actually unable to process commands due to firmware hangs or communication timeouts.

    Key detection capabilities include:

    • Driver/Firmware communication failures: Command timeouts (e.g., timeout. Will cause a leak of a command resource) and firmware hangs.
    • Hardware health check failures: Errors like health poll failed or unrecoverable hardware errors.
    • PCIe events: Surfaced via mlx5_core (e.g., insufficient power).
    • Thermal and power issues: High temperature warnings and insufficient power.
    • Network watchdog timeouts: TX queue stalls (e.g., mlx5e_tx_timeout).

  12. What is a drain plugin and how does it work

    main

    A drain plugin is a custom Kubernetes controller that replaces the default pod eviction logic in NVSentinel's node-drainer. This allows teams to coordinate node draining with custom policies (e.g., Slurm, NVCF, or specific workload managers) instead of immediate pod eviction.

    Workflow

    When customDrain.enabled=true is set in the node-drainer Helm values, the process follows these steps:

    1. Trigger: A health event occurs.
    2. Creation: node-drainer renders a drain-template (provided via a ConfigMap) into a DrainRequest custom resource in the nvsentinel namespace.
    3. Reconciliation: Your plugin (the controller) watches for DrainRequest objects, performs the necessary draining logic, and updates the status.
    4. Completion: Your plugin sets the condition status.conditions[DrainComplete]=True.
    5. Cleanup: node-drainer detects the completion condition, marks the drain as finished, and deletes the DrainRequest.

    Requirements to ship

    • A Controller: A Kubernetes controller that reconciles DrainRequest and sets DrainComplete=True.
    • A drain-template ConfigMap: A template used by node-drainer to render the DrainRequest.
    • Helm Configuration: Set customDrain.enabled=true in your node-drainer values.