HIP Documentation

HIP is a C++ Runtime API and Kernel Language designed to create portable applications for both AMD and NVIDIA GPUs from a single source code. It allows developers to write code once and run it on either platform with minimal performance impact.

Key capabilities:

• Single-source C++: Supports templates, C++11 lambdas, classes, and namespaces.
• Portability: Use the hipcc compiler driver to target either NVIDIA (via nvcc) or AMD (via HIP-Clang) platforms.
• CUDA Porting: Use HIPIFY tools to automatically convert CUDA source to HIP.
• Performance: HIP is a thin layer with little to no overhead compared to native CUDA.

HIP graphs provide a way to execute GPU tasks by predefining operations (nodes) and their dependencies (edges). This approach reduces the overhead associated with repeatedly launching kernels via standard streams, which is especially beneficial for workloads involving many short-running kernels or when using high-level frameworks that introduce redirection layers.

Supported node types (specified by hipGraphNodeType) include:

• Empty nodes
• Nested graphs
• Kernel launches
• Host-side function calls
• HIP memory functions (e.g., copy, memset)
• HIP events
• Signalling or waiting on external semaphores.

GPU kernel performance is limited by one of three main bottleneck categories. Identifying the bottleneck is the first step in optimization:

• Compute-bound: The kernel is limited by arithmetic throughput (the arithmetic bandwidth of compute units).
• Memory-bound: The kernel is limited by memory bandwidth (the rate of data movement between High Bandwidth Memory (HBM) and on-chip caches or Local Data Share (LDS)).
• Overhead-bound: The kernel is limited by latency, such as host-side scheduling, kernel launch overhead, or small array operations.

Use the Roofline model to distinguish between compute-bound and memory-bound kernels.

Peak rate is the theoretical maximum throughput (the 'speed of light') of the GPU architecture. It assumes ideal conditions where all compute units are active and pipelines are perfectly fed.

Key theoretical maximums include:

• Peak FLOPS: Maximum floating-point operations per second.
• Peak bandwidth: Maximum memory throughput.
• Peak instruction rate: Maximum instructions per cycle.

Actual performance is always lower than peak due to resource utilization, memory inefficiencies, control flow divergence, and synchronization overhead.

Reduction is a parallel programming operation that takes a range of input elements and applies a binary operation to reduce them into a single value or a shorter array.

Key concepts include:

• Identity (or Neutral) Element: A 'zero' element used to bootstrap the algorithm. It is an operand that does not change the result of the binary operation (e.g., 0 for addition, 1 for multiplication).
• Binary Operator: The operation applied to elements (e.g., std::accumulate or std::reduce in C++).
• Parallelization: To enable parallel processing, multiple identity elements can be inserted into the range, allowing partial results from different execution threads to be combined later.

MIOpen provides GPU-accelerated primitives for neural network operations (convolutions, normalizations, activations, etc.). It abstracts low-level GPU programming by providing:

• Declarative fusion: Automatically combining operations (e.g., Convolution $\rightarrow$ Bias $\rightarrow$ ReLU) into a single optimized kernel to reduce memory bandwidth pressure.
• Automatic algorithm selection: Choosing between direct, FFT-based, Winograd, or GEMM-based implementations at runtime based on tensor dimensions, filter sizes, and GPU architecture (e.g., CDNA vs RDNA).
• Memory optimization: Keeping intermediate results in LDS (Local Data Share) to minimize HBM (High Bandwidth Memory) traffic.

GPUs are designed to exploit the parallelism of 2D computational grids. For tasks like matrix multiplication, image processing, or fluid dynamics, you can achieve significant acceleration by designing kernels that operate on two-dimensional thread blocks and memory layouts.

Key Considerations for 2D Problems:

• Spatial Locality: Adjacent elements in the grid often have strong spatial correlations; optimizing memory access patterns and cache utilization is critical for performance.
• Data Representation: Many datasets (images, matrices, physical fields) map directly to a 2D coordinate space, making them natural candidates for 2D kernels.
• Hardware Alignment: Designing kernels with 2D thread blocks aligns the computation with the GPU's hardware topology, maximizing throughput and minimizing latency.