DOCSEARCHbyAlgolia
Submit Docs

Arm Compute Library

repository·main·Indexed 25 days ago

https://github.com/arm-software/computelibrary

A high-performance collection of low-level machine learning functions optimized for Arm Cortex-A, Neoverse, Mali GPUs, and x86. It provides ML primitives supporting FP32, FP16, INT8, UINT8, and BFLOAT16, featuring over 100 functions including GeMM, Winograd, and FFT. The library includes tools like Gemm Tuner for OpenCL GEMM kernel optimization and report_model_ops.py for extracting operators from TfLite models to reduce binary size.

Tokens
15.7K
Snippets
33
Records
70
Agent score
85%

What's inside Compute Library

  1. Overview of KleidiAI

    main

    KleidiAI is an open-source library providing optimized performance-critical routines, known as micro-kernels, for artificial intelligence (AI) workloads on Arm® CPUs. It is designed for easy integration into C or C++ machine learning (ML) and AI frameworks.

    Key characteristics:

    • No dependencies on external libraries.
    • No dynamic memory allocation, memory management, or scheduling.
    • Stateless, stable, and consistent API.
    • Standalone usage: Developers can include only the specific .c and .h files for the micro-kernels they need, along with the common kai_common.h header.
    • Flexible API: Designed to allow callers to control workload dispatching (e.g., processing specific sections of an output tensor) to support multi-threading or partial processing.

  2. Overview of Compute Library

    main
    Compute Library is a collection of low-level machine learning functions optimized for Arm® Cortex®-A, Arm® Neoverse™ and Arm® Mali™ GPU architectures. It provides high-performance ML primitives with support for multiple data types including FP32, FP16, INT8, UINT8, and BFLOAT16. The library features over 100 machine learning functions and supports various convolution algorithms such as GeMM, Winograd, FFT, and Direct/indirect-GeMM.

  3. Overview of Gemm Tuner

    main

    Gemm Tuner is a set of tools designed to tune the performance of OpenCL GEMM (General Matrix Multiply) kernels. It evaluates three different implementation strategies:

    1. native: Uses CLGEMMMatrixMultiplyNativeKernel.
    2. reshaped_rhs_only: Uses CLGEMMMatrixMultiplyReshapedOnlyRHSKernel.
    3. reshaped: Uses CLGEMMMatrixMultiplyReshapedKernel.

    The tuning process identifies the best kernel type and the optimal configuration (GEMMConfig) for a given set of matrix shapes (GEMMParam).

  4. Overview of KleidiAI directory structure

    main

    KleidiAI is organized into several key directories to support development, testing, and usage:

    • kai: Core library sources. Contains kai_common.h (shared utilities) and ukernels/ (micro-kernel implementations like matmul, dwconv, etc.).
    • examples: Standalone C++ sample applications demonstrating end-to-end usage of specific micro-kernels.
    • docs: Authoritative how-to guides, deep dives, and integration notes.
    • benchmark: Google Benchmark harness and matmul/ registry for comparing micro-kernel variants.
    • cmake: Toolchain definitions and helper modules for the CMake build system.
    • docker: Container files for cross-compilation environments.
    • test: Reference implementations and GoogleTest drivers for functional validation.
    • tools: Pre-commit tooling and Python requirements.

  6. Summary of KleidiAI and MLAS integration mechanics

    main

    The integration of KleidiAI into MLAS follows a specific execution pattern consisting of several stages. This pattern can be extended to other MLAS APIs such as MlasGemmPackB or MlasConv by implementing a similar override, fallback, and execution structure.

    StageDescription
    Dispatch CheckConditional on platform struct function pointer
    Pre-conditionsMatrix sizes, transpose modes, SME support
    FallbacksRecursive call into MLAS if unsupported
    Data PackingBoth LHS and RHS packed using KleidiAI routines
    Tile DispatchMulti-threaded tile-wise matmul execution
    Output Writebackmemcpy or loop with alpha/beta scaling

  7. Understand Matmul and Indirect Matmul micro-kernels

    main

    The kleidiai library provides two main types of matrix-multiplication (matmul) micro-kernels:

    1. Matmul micro-kernels: These operate directly on matrices stored in memory buffers. For optimal performance, these buffers are typically first processed into a more efficient layout using packing micro-kernels.
    2. Indirect Matmul (_imatmul) micro-kernels: These operate on indirection buffers, which are matrices containing pointers to the actual data.

  8. Understand imatmul (Indirect Matmul) concepts

    main

    Unlike standard matmul, an imatmul (indirect matrix multiplication) requires the K dimension to be evenly divided into chunks.

    Key parameters:

    • k_chunk_count: The number of chunks in the K dimension.
    • k_chunk_length: The number of elements in each chunk.

    In imatmul, the left-hand side (LHS) operand is not a table of values, but an indirection table (a table of pointers) where each row refers to a column of m_step chunks.

  10. Build int4 matmul for Android™ using NDK

    main

    To build the project for Android™, use the Android NDK toolchain with CMake. Ensure you pass the correct ANDROID_ABI and enable the i8mm feature in both CMAKE_C_FLAGS and CMAKE_CXX_FLAGS.

    Required environment variable:

    • NDK_PATH: Path to your Android NDK installation.

    Required CMake flags:

    • -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake
    • -DANDROID_ABI=arm64-v8a
    • -DANDROID_PLATFORM=android-23
    • -DCMAKE_C_FLAGS=-march=armv8.2a+i8mm
    • -DCMAKE_CXX_FLAGS=-march=armv8.2a+i8mm
    mkdir build && cd build

export NDK_PATH="your-android-ndk-path"

cmake -DCMAKE_TOOLCHAIN_FILE=${NDK_PATH}/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-23 -DCMAKE_C_FLAGS=-march=armv8.2a+i8mm -DCMAKE_CXX_FLAGS=-march=armv8.2a+i8mm ..

make

  11. Build KleidiAI matmul_clamp_f32_qsi8d32p_qsi4c32p for Android™

    main

    To build the matmul_clamp_f32_qsi8d32p_qsi4c32p example for an Android target, use the Android NDK toolchain file, specifying the target ABI (e.g., arm64-v8a) and platform version.

    $ mkdir -p build && cd build
$ cmake -DCMAKE_TOOLCHAIN_FILE=/path/to/android-ndk/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=30  -DCMAKE_BUILD_TYPE=Release ../

  12. Install and configure the Half-Precision Library

    main

    The library is a C++ header-only library. To use it, simply include half.hpp in your project. No building or linking is required.

    While fully C++98-compatible, you can explicitly enable or disable C++11 features using preprocessor symbols if automatic detection fails or if you need to disable them for specific environments:

    • HALF_ENABLE_CPP11_LONG_LONG: Enables long long for mathematical functions.
    • HALF_ENABLE_CPP11_STATIC_ASSERT: Enables static assertions for compile-time checks.
    • HALF_ENABLE_CPP11_CONSTEXPR: Enables generalized constant expressions.
    • HALF_ENABLE_CPP11_NOEXCEPT: Enables noexcept exception specifications.
    • HALF_ENABLE_CPP11_USER_LITERALS: Enables user-defined literals (e.g., 1.0_h).
    • HALF_ENABLE_CPP11_TYPE_TRAITS: Enables <type_traits> features.
    • HALF_ENABLE_CPP11_CSTDINT: Enables <cstdint> special integer types.
    • HALF_ENABLE_CPP11_CMATH: Enables improved single-precision <cmath> implementations.
    • HALF_ENABLE_CPP11_HASH: Enables std::hash support.