oneAPI Level Zero

The System Resource Tracker is a validation layer checker designed to detect memory leaks and monitor resource lifecycle in oneAPI Level Zero applications. It provides both real-time validation warnings and post-execution analysis via CSV data that can be visualized.

Key Capabilities

• Memory Leak Detection: Identifies resources (contexts, queues, modules, kernels, etc.) that are not properly destroyed by matching create/destroy API pairs. It flags destroy operations where memory increased by more than 100KB.
• Resource Tracking: Monitors 11 resource types: contexts, queues, modules, kernels, event pools, command lists, events, fences, images, samplers, and memory allocations.
• Performance & Optimization: Tracks system-level metrics (VmSize, VmRSS, VmData) and API call frequency to identify excessive allocations or memory usage patterns.

Use Cases

• Debugging: Correlate application behavior with resource usage patterns and leak events.
• Benchmarking: Compare resource usage and leak counts across different code versions.
• CI/CD Integration: Automated tracking of resource usage in test pipelines.
• Capacity Planning: Understand peak resource requirements for production workloads.

Enable ZEL_ENABLE_BASIC_LEAK_CHECKER to track Create and Destroy calls for specific handle types. The layer will report if a Create call is missing a corresponding Destroy call.

----------------------------------------------------------------------
               zeContextCreate = 1     \--->        zeContextDestroy = 1
          zeCommandQueueCreate = 1     \--->   zeCommandQueueDestroy = 1
                zeModuleCreate = 1     \--->         zeModuleDestroy = 1
                zeKernelCreate = 1     \--->         zeKernelDestroy = 1
              zeEventPoolCreate = 1     \--->      zeEventPoolDestroy = 1
   zeCommandListCreateImmediate = 1     |
             zeCommandListCreate = 1     \--->    zeCommandListDestroy = 1  ---> LEAK = 1
                   zeEventCreate = 2     \--->          zeEventDestroy = 2
                   zeFenceCreate = 1     \--->          zeFenceDestroy = 1
                         zeImageCreate = 0     \--->          zeImageDestroy = 0
                         zeSamplerCreate = 0     \--->        zeSamplerDestroy = 0
                       zeMemAllocDevice = 0     |
                         zeMemAllocHost = 1     |
                       zeMemAllocShared = 0     \--->               zeMemFree = 1

The Events Checker is a validation layer designed to detect potential deadlocks and improper event usage in the Level Zero API.

Key Capabilities:

• Deadlock Detection: It identifies improper event usage patterns that could lead to deadlocks and prints warning messages to assist developers in resolution.
• Event Reuse Validation: It detects if an event is being used more than once without being reset (e.g., attempting to signal the same event twice without a reset).

Internal Mechanism: The checker maintains an internal Directed Acyclic Graph (DAG) where:

• Nodes represent API calls (actions).
• Edges represent events.

It maintains a topological sort of these dependencies. If an API call attempts to insert an edge that would create a cycle in the graph, the checker identifies this as a potential deadlock, issues a warning, and prevents the edge from being inserted to maintain the DAG structure.

The tracker uses a post-analysis approach (via plotting scripts) and a real-time approach (via the validation layer) to identify leaks.

Leak Detection Algorithm

1. Pair Matching: The tracker matches creation and destruction API pairs (e.g., zeContextCreate ↔ zeContextDestroy).
2. Threshold Check: It flags destroy operations where memory increased by more than 100KB.
3. Cumulative Calculation: It calculates the total leaked memory across the entire program execution.
4. Visual Indicators: In generated plots, leak events are highlighted with red visual indicators and scatter markers.

Example Leak Detection Output

=== MEMORY LEAK DETECTION ===
⚠️  LEAKS DETECTED!
Total leaked memory: 0.19 MB (192.00 KB)
Number of leak events: 1

Leak events by API:
  zeEventPoolDestroy: 1 events, 0.19 MB total, 0.19 MB avg

Note: Small memory increases (< 100KB) during destroy operations may be false positives caused by internal driver caching or memory allocation granularity.