nvidia_gpu_exporter

On GPUs with MIG mode enabled, the NVML backend automatically exports per-MIG-instance metrics. No additional configuration is required for inventory and memory, but activity metrics require Hopper generation or later GPUs.

MIG Metrics:

• nvidia_smi_mig_info{uuid, mig_uuid, gpu_instance_id, compute_instance_id, profile}: Identity of each MIG device (constant 1).
• nvidia_smi_mig_memory_{total,used,free,reserved}_bytes{uuid, gpu_instance_id}: Memory metrics for the GPU instance.
• nvidia_smi_mig_{graphics_activity,sm_activity,sm_occupancy,tensor_activity}_ratio{uuid, gpu_instance_id}: Activity ratios (Hopper+ only).
• nvidia_smi_mig_pcie_throughput_{tx,rx}_bytes_per_second{uuid, gpu_instance_id}: PCIe throughput (Hopper+ only).

Attribution: Activity and throughput are reported per gpu_instance_id. To map these back to specific MIG devices, join with nvidia_smi_mig_info using (uuid, gpu_instance_id).

Process Attribution: To include gpu_instance_id and compute_instance_id labels in per-process metrics, use the --collect.compute-apps-mig flag (requires --collect.compute-apps and the NVML backend).

The exporter provides several core metrics for monitoring GPU health and utilization. Most per-GPU metrics are labeled with a uuid, which allows you to join them with the nvidia_smi_gpu_info metric. This identity metric contains static metadata such as name, driver_version, cuda_version, pci_bus_id, and serial.

Key core metrics include:

• nvidia_smi_utilization_gpu_ratio: GPU utilization as a percentage.
• nvidia_smi_memory_used_bytes: Memory currently in use (in bytes).
• nvidia_smi_power_draw_watts: Current power draw in Watts.
• nvidia_smi_temperature_gpu: GPU temperature.

# HELP nvidia_smi_gpu_info A metric with a constant '1' value labeled by gpu uuid, name, driver_model_current, driver_model_pending, vbios_version, driver_version, pci_bus_id, serial, compute_cap, pci_sub_device_id, index, cuda_version.
# TYPE nvidia_smi_gpu_info gauge
nvidia_smi_gpu_info{uuid="df6e7a7c-7314-46f8-abc4-b88b36dcf3aa",name="NVIDIA GeForce RTX 2080 SUPER",driver_version="471.11",cuda_version="11.4",pci_bus_id="00000000:71:00.0",index="0",...} 1

The --collect.backend=nvml backend reads metrics directly from the driver library (libnvidia-ml.so.1) instead of executing nvidia-smi. This is an experimental feature available in specific -nvml release artifacts.

Advantages:

• Efficiency: Spawns no processes; collection is cheaper than nvidia-smi.
• No binary required: Only requires the NVIDIA driver library (provided by NVIDIA container runtime).
• Exclusive Metrics: Provides access to driver-only families like energy_joules_total, PCIe throughput (via --collect.pcie-throughput), MIG-specific metrics, and XID error counters.

Limitations and Constraints:

• Platform: Linux x86_64, glibc-based systems only.
• Artifacts: Requires using -nvml flavored images/binaries.
• Isolation: Unlike the exec backend, a wedged driver call cannot be easily killed by the exporter.
• Incompatibility: Cannot be used with --nvidia-smi-command or remote scraping via SSH.
• Status Reporting: Failures are reported via nvidia_smi_nvml_return_code (0 for success, -1 for abandoned/failed).

nvidia_gpu_exporter --collect.backend nvml

When creating or modifying Grafana panels for nvidia_gpu_exporter, follow these technical conventions to avoid silent failures or incorrect data rendering:

• Legend Enrichment Joins: Use parentheses around the expression to ensure correct operator precedence when joining with nvidia_smi_gpu_info. Example: (expr) * on(uuid) group_left(index, name) nvidia_smi_gpu_info{...}
• Table Targets: Always include an absence sentinel using an OR arm like ... nvidia_smi_gpu_info * 0 - 1. This ensures that if a series is absent, the join doesn't drop entire columns. The sentinel must be aggregated to match the labels of the left side.
• MIG (Multi-Instance GPU) Joins: To avoid hard-errors when joining on GPUs with heterogeneous compute instances, strip the compute-instance prefix using label_replace before aggregating.
• NVML-only Families: Use an unless-guarded uuid-only arm to prevent panels from going blank if a collection fails.
• State Timelines: Use fixed/text color mode without a thresholds key; use value mappings to define band colors.
• Multi-GPU Timeseries: Use the palette-classic palette, as dynamically-named series do not support custom palettes.

To run the exporter in Docker, you must have the NVIDIA driver and the NVIDIA Container Toolkit installed on the host. The container image does not bundle NVIDIA components; instead, the toolkit injects the GPU devices, driver libraries, and the nvidia-smi binary at runtime.

Use the --gpus all flag to enable NVIDIA integration and set NVIDIA_DRIVER_CAPABILITIES=utility to indicate the container only requires the nvidia-smi/NVML tier.

Note: Without --gpus, a device reservation, or runtime: nvidia, the container will run on the default runtime without GPU access. In this state, the exporter will only serve health metrics, and nvidia_smi_last_collect_success will be 0.

docker run -d \
  --name nvidia_gpu_exporter \
  --restart unless-stopped \
  --gpus all \
  -e NVIDIA_DRIVER_CAPABILITIES=utility \
  -p 9835:9835 \
  utkuozdemir/nvidia_gpu_exporter:latest

The development stack includes a mechanism to test Prometheus alert rules. The render-rules.sh script renders the Helm chart's PrometheusRule into prometheus/rules/ with all rules force-enabled and for: durations shortened to 30s for rapid evaluation.

Testing specific alerts

• To test rule changes: Re-run ./hack/compose/render-rules.sh and then restart Prometheus: docker compose restart prometheus prometheus-demo.
• To simulate a failing exporter: Use the broken profile: docker compose --profile broken up -d exec-broken.
• To simulate a stale collection: Edit a machine file (e.g., machines/consumer.yaml) and add exit: 15 at the top level.
• To simulate a slow collection: Add delay: 2s to the machine configuration.
• To simulate thermal/power throttling: Flip clocks_event_reasons.sw_thermal_slowdown or .hw_power_brake_slowdown from 0 to 1 in the machine YAML.