DOCSEARCHbyAlgolia
Submit Docs

oneAPI Deep Neural Network Library (oneDNN)

repository·main·Indexed 26 days ago

https://github.com/uxlfoundation/onednn

An open-source, cross-platform performance library providing optimized building blocks for deep learning applications. It is designed for Intel 64/AMD64 and Arm 64-bit (AArch64) processors, as well as Intel Graphics, with experimental support for NVIDIA GPU, AMD GPU, OpenPOWER, IBMz, and RISC-V architectures. oneDNN is utilized by frameworks such as PyTorch, TensorFlow, ONNX Runtime, llama.cpp, and OpenVINO.

Tokens
134.8K
Snippets
118
Records
835
Agent score
88%

What's inside oneDNN

  1. Overview of oneDNN Functional API Examples

    main

    The Functional API provides access to individual oneDNN primitives. Examples are categorized by use case:

    Fundamental Concepts

    • Getting started with C++ basics.
    • Memory format propagation.
    • Cross-engine reordering.

    Interoperability

    • SYCL interop using Buffers or USM.
    • GPU OpenCL interop.

    Matrix Multiplication (MatMul)

    • Basic operations.
    • Quantization flavors (f8, int8, MXFP).
    • Advanced usages (COO, CSR, Grouped MatMul, Weight Compression, and Host Scalar Scale).

    Neural Network Workflows

    • CNN: Inference (f32, int8) and Training (f32, bf16).
    • RNN: Inference (f32, int8) and Training (f32).
    • RNN Architectures: Vanilla RNN, LSTM, LBR GRU, and AUGRU.

    Individual Primitives

    • Convolution: Standard and Deconvolution.
    • Linear: Inner product.
    • Pooling/Sampling: Pooling and Resampling.
    • Normalization: Batch, Group, Layer, and LRN.
    • Activation: Element-wise, PReLU, and Softmax.
    • Tensor Operations: Binary, Concatenation, Reduction, Sum, and Shuffle.
    • Memory: Reorder operations.

  2. Overview of oneDNN Graph API Examples

    main

    The Graph API allows defining computational graphs with optimization and fusion capabilities. Available examples include:

    Getting Started

    • CPU, SYCL, and GPU (OpenCL) getting started guides.

    Advanced Usage

    • CPU inference with int8 precision.
    • Single operator partitioning for both CPU and SYCL.

  3. Overview of Synthdnn

    main

    Synthdnn is a suite of scripts designed for collecting and analyzing oneDNN performance using randomly generated data. It follows a three-stage data pipeline:

    1. Synthetic problem generation: Creating test cases.
    2. Data collection: Running benchmarks and gathering results.
    3. Data analysis: Analyzing collected data.

    The primary interface is the synthdnn.py script.

  4. Overview of Pooling Primitives

    main

    The pooling primitive performs forward or backward max or average pooling operations on 1D, 2D, or 3D spatial data.

    Average Pooling Algorithms

    Average pooling supports two specific algorithms:

    • dnnl_pooling_avg_include_padding: The denominator ($DENOM$) is the kernel size ($KH \cdot KW$).
    • dnnl_pooling_avg_exclude_padding: The denominator ($DENOM$) is the size of the overlap between the averaging window and the image.

    Forward vs. Backward Propagation

    • Forward Inference: Does not require a workspace for max pooling.
    • Forward Training: Max pooling requires a workspace to save indices where a maximum was found. This workspace can be created via workspace_desc() from the pooling primitive descriptor.
    • Backward: Computes the gradient of the source ($\text{diffsrc}$) based on the gradient of the destination ($\text{diffdst}$) and, in the case of max pooling, the workspace used during the forward pass.

  5. Overview of oneDNN Microkernel (uKernel) API Examples

    main
    The oneDNN microkernel API is a low-level abstraction for CPU that provides maximum flexibility. It allows users to maintain full control over threading logic, blocking logic, and code customization with minimal overhead.

  6. Overview of oneAPI Deep Neural Network Library (oneDNN)

    main

    oneAPI Deep Neural Network Library (oneDNN) is an open-source, cross-platform performance library providing basic building blocks for deep learning applications. It is an implementation of the oneAPI specification and is optimized for:

    • Intel 64/AMD64 architecture-based processors
    • Arm(R) 64-bit Architecture (AArch64)-based processors
    • Intel Graphics

    It also provides experimental support for NVIDIA* GPU, AMD* GPU, OpenPOWER* Power ISA (PPC64), IBMz* (s390x), and RISC-V architectures.

    Deep learning practitioners should use applications that have oneDNN enabled, such as PyTorch, TensorFlow, ONNX Runtime, llama.cpp, and OpenVINO.

  7. Understand GPU JIT-compiled Kernels in oneDNN

    main
    oneDNN uses JIT (Just-In-Time) compilation for GPU kernels via the nGEN library. Instead of writing kernels in OpenCL C and relying on the OpenCL runtime to compile them to assembly, oneDNN uses nGEN to emit Gen assembly directly. This approach aims for assembly-level performance, broad coverage of data types and hardware, and better generalization through common building blocks.

  8. Understand the Inner Product primitive

    main

    The inner product primitive (also known as fully connected) treats each activation in a minibatch as a vector and computes its product with a 2D weights tensor, producing a 2D output tensor.

    Forward Propagation Formula: dst(n, oc) = bias(oc) + sum_{ic=0}^{IC-1} src(n, ic) * weights(oc, ic)

    If src and weights have spatial dimensions (e.g., 4D tensors), they are flattened to 2D. For 4D tensors, the spatial dimensions must match (e.g., KH = IH and KW = IW).

    Backward Propagation:

    • Computes diffsrc based on diffdst and weights.
    • Computes diffweights and diffbias based on diffdst and src (weights update).

  9. Understand the oneDNN programming model

    main

    The oneDNN programming model is based on executing Primitives to process data stored in Memory Objects. This execution occurs on an Engine within the context of a Stream.

    Key entities include:

    • Primitives: Objects encapsulating specific computations (e.g., convolution, LSTM). They can store immutable state (parameters, cache blocking) and mutable state (scratchpad memory used for temporary storage).
    • Engines: Abstractions of computational devices like CPUs or GPUs.
    • Streams: Execution contexts tied to a specific engine (e.g., OpenCL command queues).
    • Memory Objects: Handles to memory on a specific engine, containing tensor dimensions, data types, and memory formats.

  10. Understand the oneDNN ukernel API programming model

    main

    The oneDNN micro-kernel API (ukernel API) is a low-level, sequential abstraction designed specifically for CPUs. It provides maximum flexibility by giving the user full control over:

    • Threading logic: The API is sequential and independent of any threading runtime.
    • Blocking logic: Users configure ukernel object sizes to optimize for local caches.
    • Customization: Users can interleave custom code with ukernel code within parallel regions.

    The API uses a minimal number of abstractions to ensure low overhead.

  11. Understand the oneDNN XPU implementation structure

    main

    The XPU implementation in oneDNN is designed to be vendor-agnostic and organized around heterogeneous runtimes.

    • Common XPU code: Resides in the xpu/ directory and is shared across different runtimes.
    • Runtime-specific code: Resides in sub-directories under xpu/ (e.g., xpu/sycl/ for SYCL or xpu/ocl/ for OpenCL).

    When developing or extending the library, vendor-specific code should access the common XPU code, but common XPU code should avoid accessing vendor-specific code whenever possible to maintain abstraction.

  12. Understand the oneDNN GPU implementation structure

    main

    The oneDNN GPU implementation is organized by vendor to ensure code independence.

    • Common GPU code: Resides in the gpu/ directory.
    • Vendor-specific code: Resides in gpu/<vendor>/ sub-directories (e.g., gpu/intel/, gpu/nvidia/, gpu/amd/).
    • Kernel language code: Vendor sub-directories may contain further sub-directories for specific kernel languages (e.g., gpu/intel/sycl/).
    • Generic code: The gpu/generic/ directory contains vendor-agnostic code, such as generic SYCL kernels.

    Note: XPU-specific code resides outside the gpu/ directory but is utilized by GPU-specific code. For details on XPU, see the oneDNN XPU Implementation documentation.