NVIDIA Data Center GPU Manager (DCGM)

DCGM follows Semantic Versioning (SemVer) in the format X.Y.z. The release cycle is tied to the driver LLB lifecycle, with major or minor revisions (X or Y) occurring approximately twice a year.

Release Types:

• Current Release: The primary branch for new features and patch releases. This is the recommended version for most users.
• Maintenance Release: A supported version that is eligible for patch releases. However, if a fix is available in the Current Release, users are encouraged to upgrade to that instead of waiting for a maintenance patch.
• Legacy Releases: Versions that have passed their one-year support window. No patch releases are created for legacy releases. Bugs reported in legacy versions will be addressed in the Current or Maintenance releases instead.

Versioning Rules:

• Major (X): Breaking API changes (often occurring at architecture boundaries).
• Minor (Y): New API additions.
• Patch (z): Bug fixes (released quarterly).

NVML Injection allows you to mock the return values of NVML calls for testing purposes. There are two ways to provide mocked values:

1. YAML Configuration: Loading values from an NVML injection YAML file (e.g., via run_with_injection_nvml_using_specific_sku).
2. Manual Injection: Using the dcgmInjectNvmlDevice API.

Priority Rule: Manual injection via dcgmInjectNvmlDevice has higher priority than values loaded from a YAML file. If both are present, the manually injected value is used.

Key Implementation Rules

• Target Naming: When specifying the injection target, do not include the Get prefix. For example, to mock nvmlDeviceGetFanSpeed, use the target name FanSpeed. This allows the same injection to affect both getters and setters (e.g., nvmlDeviceSetFanSpeed).
• Device Handles: Since the Python injection code cannot directly provide a device handle, pass the gpuId. The dcgmInjectNvmlDevice function will automatically translate the gpuId into the appropriate device handle.
• Overriding: Calling dcgmInjectNvmlDevice multiple times on the same API will override previous values. To test multiple fields, perform individual injections.

# Example of mocking nvmlDeviceGetNvLinkErrorCounter
def mock_nvlink_error_counter(handle, gpuId, linkId, counterType, nvmlRet, value):
    # 1. Define the return value (injectedRet)
    injectedRet = nvml_injection.c_injectNvmlRet_t()
    injectedRet.nvmlRet = nvmlRet
    injectedRet.values[0].type = nvml_injection_structs.c_injectionArgType_t.INJECTION_ULONG_LONG
    injectedRet.values[0].value.ULongLong = value
    injectedRet.valueCount = 1

    # 2. Define extra keys (parameters) for the function
    extraKeysType = nvml_injection_structs.c_injectNvmlVal_t * 2
    extraKeys = extraKeysType()
    extraKeys[0].type = nvml_injection_structs.c_injectionArgType_t.INJECTION_UINT
    extraKeys[0].value.UInt = linkId
    extraKeys[1].type = nvml_injection_structs.c_injectionArgType_t.INJECTION_NVLINKERRORCOUNTER
    extraKeys[1].value.NvLinkErrorCounter = counterType

    # 3. Execute injection
    ret = dcgm_agent_internal.dcgmInjectNvmlDevice(handle, gpuId, "NvLinkErrorCounter", extraKeys, 2, injectedRet)
    assert (ret == dcgm_structs.DCGM_ST_OK)

The YAML file maps NVML function calls to specific return values using a key-based system.

Keys and Extra Keys

• Key: The key is the suffix of the NVML function name. For example, nvmlDeviceGetTotalEccErrors uses the key TotalEccErrors. This allows setters and getters (e.g., nvmlDeviceGetFanSpeed and nvmlDeviceSetFanSpeed) to share the same state.
• Extra Keys: For functions with multiple inputs, InjectionArgument (a union of NVML structures and basic types) is used. For example, a function with a fan parameter might use INJECTION_UINT as an extra key.

YAML Structure Example

Each entry maps a key (and optional extra keys) to a FunctionReturn (nvmlReturn_t) and a ReturnValue.

APIRestriction:
  0:
    FunctionReturn: 3
  1:
    FunctionReturn: 0
    ReturnValue: 0

In this example, APIRestriction is the primary key, and 0 and 1 are extra keys representing different input arguments.

To enable injection for a new NVML function, follow these steps:

1. Define the function: Add the new functions and definitions to dcgm_nvml.h and entry_points.h.
2. Generate stubs: Run generate_nvml_stubs.py from the dcgm/ root to update the fake implementations.
3. Update the recorder: Update nvml_api_recorder.py so it can capture values from the new function. This involves:
   • Adding the function to _nvml_device_attr_funcs (for simple functions) or _nvml_device_extra_key_attr_funcs (for functions with multiple inputs).
   • Writing a YAML serializer (e.g., pci_info_parser) to convert the C++ structure into a dictionary. Note: Use C++ field names as the serialized keys in the YAML.
   • For functions using references for output (e.g., nvmlDeviceGetGpuFabricInfo), write a wrapper function in nvml_api_recorder.py to handle the output parameter.
4. Update YAML: Add the new key to your test YAML files or modify the YAML directly to simulate specific return values.
5. Test: Run your program with NVML_INJECTION_MODE=True and NVML_YAML_FILE=<path>.