NVIDIA DCGM Exporter

To map GPUs to High-Performance Computing (HPC) jobs, configure your environment to generate mapping files.

File Format:

• Filename: Unique GPU ID or GPU ID.MIG ID (e.g., 0, 1, 2.0).
• Content: One or more JOB IDs per line.

Enabling in DCGM-Exporter: Use the --hpc-job-mapping-dir flag or the DCGM_HPC_JOB_MAPPING_DIR environment variable to point to the directory containing these files.

If you enable profiling by adding --enable-pprof to the arguments list, you must also configure security to prevent exposing sensitive runtime profiling data. The exporter will fail to start if pprof is enabled without a web configuration file.

To secure it, enable tlsServerConfig and/or configure basicAuth.users. The chart will then automatically mount the exporter-toolkit web config and set DCGM_EXPORTER_WEB_CONFIG_FILE.

The recommended way to install DCGM-Exporter on Kubernetes is using the official Helm chart.

Note: NVIDIA recommends using the NVIDIA GPU Operator instead of installing DCGM-Exporter directly for production environments.

# Add the Helm repository
helm repo add gpu-helm-charts \
  https://nvidia.github.io/dcgm-exporter/helm-charts

# Update the repository
helm repo update

# Install the chart
helm install \
    --generate-name \
    gpu-helm-charts/dcgm-exporter

To gather GPU metrics on a local node using Docker, run the dcgm-exporter container with the necessary capabilities and GPU access.

After starting the container, you can verify the metrics by querying the /metrics endpoint on port 9400.

docker run -d --gpus all --cap-add SYS_ADMIN --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless

# Check the metrics
curl localhost:9400/metrics

Deploy NVIDIA DCGM Exporter to your Kubernetes cluster using the provided Helm chart. You can install it with default settings or provide a custom values file for specific configurations like debug dumps or custom metrics.

# Install with default configuration
helm install dcgm-exporter ./deployment

# Install with custom values (create your own values file)
helm install dcgm-exporter ./deployment -f my-debug-values.yaml

Debug dumps allow dcgm-exporter to write runtime object information (device info, metrics, runtime state) to files for troubleshooting.

Production Recommendation: Use a persistent directory like /var/log/dcgm-exporter-debug instead of the default /tmp/dcgm-exporter-debug to ensure files survive pod restarts and node reboots. The chart automatically creates a hostPath volume when debug dumps are enabled.

debugDump:
  enabled: true
  directory: "/var/log/dcgm-exporter-debug"  # Persistent location
  retention: 48  # hours
  compression: true

The package includes nvidia-dcgm-exporter.service for host deployments. To customize the service (e.g., changing restart behavior) without modifying the package-managed unit, use a systemd drop-in file in /etc/systemd/system/nvidia-dcgm-exporter.service.d/.

# /etc/systemd/system/nvidia-dcgm-exporter.service.d/restart.conf
[Unit]
StartLimitIntervalSec=0

[Service]
Restart=on-failure
RestartSec=10s

sudo systemctl daemon-reload
sudo systemctl restart nvidia-dcgm-exporter.service

To prevent scrape failures on dense GPU or MIG nodes, align the exporter's HTTP server timeouts with the Prometheus ServiceMonitor scrape budget.

Constraint: serviceMonitor.scrapeTimeout must be lower than service.webWriteTimeout and no greater than serviceMonitor.interval.

service:
  webReadTimeout: 10s
  webWriteTimeout: 30s

serviceMonitor:
  interval: 30s
  scrapeTimeout: 25s

DCGM-Exporter supports TLS and basic authentication via the exporter-toolkit. To enable these security features, you must provide a web configuration file using the --web-config-file flag.

dcgm-exporter --web-config-file=web-config.yaml