DOCSEARCHbyAlgolia
Submit Docs

pynvml

repository·master·Indexed 43 days ago

https://github.com/gpuopenanalytics/pynvml

Unofficial Python utilities for the NVIDIA Management Library (NVML) version 13.0.1. Provides high-level bindings for the nvidia-smi API via the pynvml_utils module, including the DeviceQuery method for retrieving GPU metrics, clock throttling reasons, memory capacity, and ECC error reporting.

Tokens
4.8K
Snippets
5
Records
8
Agent score
5%

What's inside pynvml

  1. Monitor GPU clock throttling reasons

    master

    To identify why GPU clocks are being reduced, query the clocks_throttle_reasons properties. If all returned values are "Not Active", the clocks are running at their maximum possible frequency. Common reasons include:

    • clocks_throttle_reasons.sw_thermal_slowdown: Software thermal capping due to high temperature.
    • clocks_throttle_reasons.hw_thermal_slowdown: Hardware thermal slowdown (core clocks reduced by factor of 2 or more).
    • clocks_throttle_reasons.sw_power_cap: Software power scaling due to power consumption limits.
    • clocks_throttle_reasons.hw_power_brake_slowdown: External power brake assertion (e.g., from system PSU).
    • clocks_throttle_reasons.sync_boost: GPU is part of a sync boost group to maximize performance per watt.

  2. Understand GPU Memory and ECC error reporting

    master

    Memory Capacity

    Query memory.total, memory.used, and memory.free. Note that total available memory is reduced if ECC is enabled due to parity bits.

    ECC Errors

    Errors are reported across two timescales:

    • Volatile: Errors detected since the last driver load.
    • Aggregate: Lifetime counters that persist indefinitely.

    Errors can be categorized by type:

    • device_memory / dram (Global device memory)
    • register_file (Register file memory)
    • l1_cache / l2_cache (Cache memory)
    • texture_memory (Texture memory)
    • cbu / sram (SRAMs)

    Errors are further classified as corrected (single-bit, automatically fixed by HW) or uncorrected (double-bit, results in data corruption).

    Retired Pages

    When memory pages become unreliable (due to multiple single-bit errors or a double-bit error), the driver can retire them. Use retired_pages.single_bit_ecc.count or retired_pages.double_bit.count to check the number of retired pages.

  3. Install pynvml via pip

    master

    To install this project, use pip from the project root. Note that this project requires nvidia-ml-py as a dependency.

    Warning: This project is deprecated. The pynvml module is no longer maintained here. This project primarily provides unofficial NVML Python utilities in the pynvml_utils module for demonstration purposes.

    pip install .

  4. Get help for DeviceQuery queries

    master

    You can use the --help-query-gpu flag within DeviceQuery to see available query options for GPUs.

    from pynvml_utils import nvidia_smi
nvsmi = nvidia_smi.getInstance()
print(nvsmi.DeviceQuery('--help-query-gpu'), end='\n')

  5. Query device metrics with DeviceQuery

    master

    Use the DeviceQuery method from an nvidia_smi instance to retrieve specific GPU metrics. The method accepts a string of comma-separated metric names (e.g., 'memory.free, memory.total').

    from pynvml_utils import nvidia_smi
nvsmi = nvidia_smi.getInstance()
print(nvsmi.DeviceQuery('memory.free, memory.total'))

  6. Use the nvidia_smi high-level API

    master

    The pynvml_utils.nvidia_smi module provides high-level bindings for the nvidia-smi API. You can access these bindings by calling getInstance() on the nvidia_smi module.

    Warning: The nvidia_smi module is intended for demonstration purposes only and has no guarantee of long-term maintenance or support.

    from pynvml_utils import nvidia_smi
nvsmi = nvidia_smi.getInstance()
# Query specific device metrics
nvsmi.DeviceQuery('memory.free, memory.total')

  7. Query additional GPU command outputs

    master

    In addition to standard property queries, use these commands to retrieve specialized lists:

    • supported-clocks: List of all supported clocks.
    • compute-apps: List of currently active compute processes.
    • accounted-apps: List of accounted compute processes.
    • clocks: Query clock information/policy.

  8. Query GPU properties by string name

    master

    You can query various GPU attributes using specific string identifiers or their corresponding NVSMI_ enumeration constants. These properties cover hardware identification, PCI details, power management, clock speeds, memory utilization, and error states.

    List of valid properties by string name : enumeration to query:

"timestamp" or NVSMI_TIMESTAMP
"driver_version" or NVSMI_DRIVER_VERSION
"count" or NVSMI_COUNT
"name" or "gpu_name" or NVSMI_NAME
"serial" or "gpu_serial" or NVSMI_SERIALNUMBER
"uuid" or "gpu_uuid" or NVSMI_UUID
"pci.bus_id" or "gpu_bus_id" or NVSMI_PCI_BUS_ID
"pci.domain" or NVSMI_PCI_DOMAIN
"pci.bus" or NVSMI_PCI_BUS
"pci.device" or NVSMI_PCI_DEVICE
"pci.device_id" or NVSMI_PCI_DEVICE_ID
"pci.sub_device_id" or NVSMI_PCI_SUBDEVICE_ID
"pcie.link.gen.current" or NVSMI_PCI_LINK_GEN_CUR
"pcie.link.gen.max" or NVSMI_PCI_LINK_GEN_MAX
"pcie.link.width.current" or NVSMI_PCI_LINK_WIDTH_CUR
"pcie.link.width.max" or NVSMI_PCI_LINK_WIDTH_MAX
"index" or NVSMI_INDEX
"display_mode" or NVSMI_DISPLAY_MODE
"display_active" or NVSMI_DISPLAY_ACTIVE
"persistence_mode" or NVSMI_PERSISTENCE_MODE
"accounting.mode" or NVSMI_ACCT_MODE
"accounting.buffer_size" or NVSMI_ACCT_BUFFER_SIZE
"driver_model.current" or NVSMI_DRIVER_MODEL_CUR
"driver_model.pending" or NVSMI_DRIVER_MODEL_PENDING
"vbios_version" or NVSMI_VBIOS_VER
"inforom.img" or "inforom.image" or NVSMI_INFOROM_IMG
"inforom.oem" or NVSMI_INFOROM_OEM
"inforom.ecc" or NVSMI_INFOROM_ECC
"inforom.pwr" or "inforom.power" or NVSMI_INFOROM_PWR
"gom.current" or "gpu_operation_mode.current" or NVSMI_GOM_CUR
"gom.pending" or "gpu_operation_mode.pending" or NVSMI_GOM_PENDING
"fan.speed" or NVSMI_FAN_SPEED
"pstate" or NVSMI_PSTATE
"clocks_throttle_reasons.supported" or NVSMI_CLOCK_THROTTLE_REASONS_SUPPORTED
"clocks_throttle_reasons.active" or NVSMI_CLOCK_THROTTLE_REASONS_ACTIVE
"clocks_throttle_reasons.gpu_idle" or NVSMI_CLOCK_THROTTLE_REASONS_IDLE
"clocks_throttle_reasons.applications_clocks_setting" or NVSMI_CLOCK_THROTTLE_REASONS_APP_SETTING
"clocks_throttle_reasons.sw_power_cap" or NVSMI_CLOCK_THROTTLE_REASONS_SW_PWR_CAP
"clocks_throttle_reasons.hw_slowdown" or NVSMI_CLOCK_THROTTLE_REASONS_HW_SLOWDOWN
"clocks_throttle_reasons.hw_thermal_slowdown" or NVSMI_CLOCK_THROTTLE_REASONS_HW_THERMAL_SLOWDOWN
"clocks_throttle_reasons.hw_power_brake_slowdown" or NVSMI_CLOCK_THROTTLE_REASONS_HW_PWR_BRAKE_SLOWDOWN
"clocks_throttle_reasons.sw_thermal_slowdown" or NVSMI_CLOCK_THROTTLE_REASONS_SW_THERMAL_SLOWDOWN
"clocks_throttle_reasons.sync_boost" or NVSMI_CLOCK_THROTTLE_REASONS_SYNC_BOOST
"memory.total" or NVSMI_MEMORY_TOTAL
"memory.used" or NVSMI_MEMORY_USED
"memory.free" or NVSMI_MEMORY_FREE
"compute_mode" or NVSMI_COMPUTE_MODE
"utilization.gpu" or NVSMI_UTILIZATION_GPU
"utilization.memory" or NVSMI_UTILIZATION_MEM
"encoder.stats.sessionCount" or NVSMI_ENCODER_STATS_SESSIONCOUNT
"encoder.stats.averageFps" or NVSMI_ENCODER_STATS_AVG_EPS
"encoder.stats.averageLatency" or NVSMI_ENCODER_STATS_AVG_LATENCY
"ecc.mode.current" or NVSMI_ECC_MODE_CUR
"ecc.mode.pending" or NVSMI_ECC_MODE_PENDING
"ecc.errors.corrected.volatile.device_memory" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_DEV_MEM
"ecc.errors.corrected.volatile.dram" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_DRAM
"ecc.errors.corrected.volatile.register_file" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_REGFILE
"ecc.errors.corrected.volatile.l1_cache" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_L1CACHE
"ecc.errors.corrected.volatile.l2_cache" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_L2CACHE
"ecc.errors.corrected.volatile.texture_memory" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_TEXTURE
"ecc.errors.corrected.volatile.cbu" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_CBU
"ecc.errors.corrected.volatile.sram" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_SRAM
"ecc.errors.corrected.volatile.total" or NVSMI_ECC_ERROR_CORRECTED_VOLATILE_TOTAL
"ecc.errors.corrected.aggregate.device_memory" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_DEV_MEM
"ecc.errors.corrected.aggregate.dram" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_DRAM
"ecc.errors.corrected.aggregate.register_file" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_REGFILE
"ecc.errors.corrected.aggregate.l1_cache" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_L1CACHE
"ecc.errors.corrected.aggregate.l2_cache" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_L2CACHE
"ecc.errors.corrected.aggregate.texture_memory" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_TEXTURE
"ecc.errors.corrected.aggregate.cbu" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_CBU
"ecc.errors.corrected.aggregate.sram" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_SRAM
"ecc.errors.corrected.aggregate.total" or NVSMI_ECC_ERROR_CORRECTED_AGGREGATE_TOTAL
"ecc.errors.uncorrected.volatile.device_memory" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_DEV_MEM
"ecc.errors.uncorrected.volatile.dram" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_DRAM
"ecc.errors.uncorrected.volatile.register_file" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_REGFILE
"ecc.errors.uncorrected.volatile.l1_cache" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_L1CACHE
"ecc.errors.uncorrected.volatile.l2_cache" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_L2CACHE
"ecc.errors.uncorrected.volatile.texture_memory" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_TEXTURE
"ecc.errors.uncorrected.volatile.cbu" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_CBU
"ecc.errors.uncorrected.volatile.sram" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_SRAM
"ecc.errors.uncorrected.volatile.total" or NVSMI_ECC_ERROR_UNCORRECTED_VOLATILE_TOTAL
"ecc.errors.uncorrected.aggregate.device_memory" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_DEV_MEM
"ecc.errors.uncorrected.aggregate.dram"  or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_DRAM
"ecc.errors.uncorrected.aggregate.register_file" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_REGFILE
"ecc.errors.uncorrected.aggregate.l1_cache" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_L1CACHE
"ecc.errors.uncorrected.aggregate.l2_cache" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_L2CACHE
"ecc.errors.uncorrected.aggregate.texture_memory" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_TEXTURE
"ecc.errors.uncorrected.aggregate.cbu" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_CBU
"ecc.errors.uncorrected.aggregate.sram" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_SRAM
"ecc.errors.uncorrected.aggregate.total" or NVSMI_ECC_ERROR_UNCORRECTED_AGGREGATE_TOTAL
"retired_pages.single_bit_ecc.count" or "retired_pages.sbe" or NVSMI_RETIREDPAGES_SINGLE_BIT_ECC_COUNT
"retired_pages.double_bit.count" or "retired_pages.dbe" or NVSMI_RETIREDPAGES_DOUBLE_BIT_ECC_COUNT
"retired_pages.pending" or NVSMI_RETIREDPAGES_PENDING
"temperature.gpu" or NVSMI_TEMPERATURE_GPU
"temperature.memory" or NVSMI_TEMPERATURE_MEMORY
"power.management" or NVSMI_POWER_MGMT
"power.draw" or NVSMI_POWER_DRAW
"power.limit" or NVSMI_POWER_LIMIT
"enforced.power.limit" or NVSMI_POWER_LIMIT_ENFORCED
"power.default_limit" or NVSMI_POWER_LIMIT_DEFAULT
"power.min_limit" or NVSMI_POWER_LIMIT_MIN
"power.max_limit" or NVSMI_POWER_LIMIT_MAX
"clocks.current.graphics" or "clocks.gr" or NVSMI_CLOCKS_GRAPHICS_CUR
"clocks.current.sm" or "clocks.sm" or NVSMI_CLOCKS_SM_CUR
"clocks.current.memory" or "clocks.mem" or NVSMI_CLOCKS_MEMORY_CUR
"clocks.current.video" or "clocks.video" or NVSMI_CLOCKS_VIDEO_CUR
"clocks.applications.graphics" or "clocks.applications.gr" or NVSMI_CLOCKS_APPL_GRAPHICS
"clocks.applications.memory" or "clocks.applications.mem" or NVSMI_CLOCKS_APPL_MEMORY
"clocks.default_applications.graphics" or "clocks.default_applications.gr" or NVSMI_CLOCKS_APPL_GRAPHICS_DEFAULT
"clocks.default_applications.memory" or "clocks.default_applications.mem" or NVSMI_CLOCKS_APPL_MEMORY_DEFAULT
"clocks.max.graphics" or "clocks.max.gr" or NVSMI_CLOCKS_GRAPHICS_MAX
"clocks.max.sm" or "clocks.max.sm" or NVSMI_CLOCKS_SM_MAX
"clocks.max.memory" or "clocks.max.mem" or NVSMI_CLOCKS_MEMORY_MAX
"mig.mode.current" or NVSMI_MIG_MODE_CURRENT
"mig.mode.pending" or NVSMI_MIG_MODE_PENDING