MLU-OPS™ Reference Repository

repository·master·Indexed 41 days ago

https://github.com/cambricon/mlu-ops

A reference repository providing example code and tools for developing high-performance custom operators for Cambricon Machine Learning Units (MLU) using C and BANG C interfaces. It includes high-quality BANG C operator implementations, performance analysis tools (Perf-Analyse), runtime test case generation (Gen-case), and GTest modules for memory leak and code coverage testing. Supports various FFT modes (rfft, irfft, fft, ifft) and algorithms including DFT, Cooley-Tukey, and Stockham.

Tokens
190.7K
Snippets
264
Records
605
Agent score
51%

What's inside MLU-OPS

  1. Overview of Cambricon MLU-OPS

    master

    Cambricon MLU-OPS is a high-performance computing library optimized for AI network scenarios on Cambricon MLU hardware. It provides a collection of common operators with high-speed optimization and offers a concise, efficient, universal, flexible, and extensible programming interface.

    Key features include:

    • Operator Development: Implemented using BANG C (Cambricon's programming language specifically developed for MLU hardware).
    • Design Principles: Focuses on ease of use, supporting various data layouts, flexible dimension constraints, and diverse data types.
    • Performance: Optimized for Cambricon hardware architecture to achieve peak performance and minimize memory footprint.
    • Resource Management: Includes interfaces for resource management to support multi-threaded and multi-card application scenarios.
  2. Overview of MLU-OPS™

    master

    MLU-OPS™ provides example code for developing high-performance operators using C interfaces based on Cambricon Machine Learning Units (MLU). It is designed as a reference for developers to create custom operators for implementing specific model computations.

    Key features include:

    • Operator accuracy and performance acceptance standards.
    • An Op List containing high-quality implementations of BANG C operators.
    • Guidance on using CNNL base operators.
    • Testing modules including GTest (supporting memory leak and code coverage testing).
    • Gen-case: A runtime test case generation tool.
    • Perf-Analyse: An operator performance analysis tool.
  3. Design specification for diff_iou_rotated_sort_vertices_forward operator

    master
    This document outlines the design specification for the diff_iou_rotated_sort_vertices_forward operator. It covers requirement analysis, interface design, implementation schemes, and performance optimization records. The operator is designed to handle specific tensor shapes and logic related to rotated IoU (Intersection over Union) calculations and vertex sorting.
  4. What is the poly_nms operator?

    master

    The poly_nms (polygon Non-Maximum Suppression) operator is designed to remove highly redundant polygon bounding boxes from a set of inputs. It is primarily used in object detection networks like FasterRCNN and trans obb to filter overlapping polygons based on an Intersection over Union (IoU) threshold.

    Key Characteristics:

    • Input Format: Expects boxes in the format [[x1, y1, x2, y2, x3, y3, x4, y4, score], ...].
    • Data Type: Supports float input.
    • Vertex Ordering: For reliable results, input box vertices must be sorted in either clockwise or counter-clockwise order. Unordered vertices may lead to inconsistent results compared to competitor implementations.
    • Handling Special Values:
      • If scores contain inf, the operator treats the box as having high priority.
      • If scores contain -inf, the operator treats the box as having low priority.
      • If scores contain nan, the operator selects the box with the larger score when an overlap occurs.
      • If boxes coordinates contain inf or nan, the operator treats that specific box as not intersecting with any other boxes.
  5. What is the prior_box operator and how does it work?

    master

    The prior_box operator is used in the SSD (Single Shot MultiBox Detector) algorithm to generate candidate bounding boxes (priors) for each point in an input feature map.

    Core Logic:

    • For every pixel in the input height $\times$ width feature map, it generates num_priors candidate boxes.
    • The total number of boxes generated is height * width * num_priors.
    • Each box is represented by coordinates (x1, y1, x2, y2), which are normalized to the range [0, 1] relative to the original image dimensions (im_width, im_height).
    • The number of priors (num_priors) is determined by the lengths of min_sizes, max_sizes, and aspect_ratios.
    • Center Point: The center of the boxes for a specific pixel is calculated using an offset and the step sizes (step_w, step_h) relative to the image dimensions.
    • Clipping: If is_clip is set to True, the coordinates are clipped to the [0, 1] range using max(coord, 0) and min(coord, 1).
    • Variance: The operator also outputs a var tensor, which replicates the input variance (a 4-element array) for every generated box to facilitate decoding.
    # Conceptual logic for box generation
    pixel_pos = GetPixelPos(width, height, 1, 4)  # shape (W, H, 1, 4)
    box_offset = GetBoxOffset(1, 1, num_priors, 4) # shape (1, 1, num_priors, 4)
    lower_bound = [0]
    upper_bound = [W, H, W, H]
    output = pixel_pos + box_offset
    output = max(lower_bound, output)
    output = min(upper_bound, output)
  6. Implementation strategy for 1D FFT

    master

    The 1D FFT implementation is optimized for MLU hardware by adapting the butterfly decomposition and network selection:

    • Butterfly Decomposition: On the host side, input scales are decomposed into large-scale bases (limited by NRAM capacity) and smaller-scale bases to reduce complexity. The algorithm prioritizes larger radices (e.g., 64) and scales down until a valid decomposition is found.
    • Network Selection: The Stockham butterfly network is used because it supports arbitrary mixed bases and provides continuous input/output without bit-reversal.
    • Butterfly Computation: To leverage MLU matrix calculation units, butterfly operations are performed using matrix multiplication. A dft_matrix containing rotation factor data and matrix sizes is generated for different bases, and rotation factors are pre-copied from GDRAM to NRAM.
  7. Implementation logic for the bbox_overlaps operator

    master

    The bbox_overlaps operator calculates the overlap (Intersection over Union - IoU) between two sets of bounding boxes. The implementation strategy depends on the aligned flag:

    • Aligned Mode (aligned: true): The vertical (batch) coordinates of the input data are copied to NRAM as vectors for vectorized computation. In this mode, the number of rows in bbox1 and bbox2 must be equal.
    • Non-aligned Mode (aligned: false): Each coordinate set in bbox1 is expanded into a vector matching the length of the coordinate vectors in bbox2 to facilitate calculation.

    Output Dimensions:

    • If aligned == true: The output ious has a total dimension of 1, where the size matches the number of rows in bbox1 or bbox2.
    • If aligned == false: The output ious has a total dimension of 2, where dimension 0 is the number of rows in bbox1 and dimension 1 is the number of rows in bbox2.
  8. Understand Task Types: Block vs Union

    master

    Task types specify the hardware resources (MLU Cores or Clusters) required by a Kernel.

    Block Tasks

    • Represents a Kernel that occupies at least one MLU Core.
    • Limitations: Does not support shared SRAM or communication between different Clusters.
    • Supported by: All Cambricon hardware.
    • Execution: If multiple cores are available, tasks are distributed across cores; otherwise, they execute serially on a single core.

    UnionN Tasks (N = 2^n)

    • Represents a Kernel that occupies at least $N$ Clusters.
    • Hardware Support: Depends on device configuration. For a device with $M$ Clusters, the maximum $N$ is $2^{\log_{2}{M}}$.
    • Union1: dim.x must be a multiple of 4. Supports 4 MLU Cores and 1 Memory Core sharing SRAM within a single Cluster. No inter-cluster communication.
    • Union2/4/8/16: dim.x must be a multiple of 8/16/32/64 respectively. Supports inter-cluster communication between 2/4/8/16 Clusters. Each Cluster's 4 MLU Cores and Memory Core share SRAM.
    • Constraint: Communication via shared SRAM is not supported between tasks with different taskIdY or taskIdZ values.
  9. Implementation design for the threeNN operator

    master

    The threeNN operator calculates the three nearest neighbors (distances and indices) for each point in an unknown set relative to a known set.

    Execution Pipeline

    1. Host-side Layout Transpose: The known data is transposed from shape (b, m, 3) to (b, 3, m) to facilitate vector operations.
    2. NRAM Segmentation: Due to memory constraints, both unknown and known datasets are processed in segments (unknown_segment and known_segment) if they exceed available NRAM.
    3. Distance Calculation: For each unknown point and known_segment, the squared distance ($dist^2$) is calculated using the following sequence of BANG instructions:
      • __bang_sub_const(): Calculate differences for $x, y, z$ coordinates.
      • __bang_square(): Square the differences.
      • __bang_add(): Sum the squared differences to get $dist^2$.
    4. Top-3 Selection: The operator identifies the three smallest distances and their corresponding indices using __bang_argmin(). If data is segmented, it performs a multi-pass comparison to find the global top-3 across all segments.

    Task Splitting (Multi-core)

    Tasks are split across MLU cores using cnrtFuncTypeUnion1. The unknown coordinate points (totaling $b \times n$) are divided uniformly into taskDim parts.