ILGPU Documentation

When designing data structures for GPU kernels in ILGPU, you can choose between two primary memory layouts: Array of Structs (AoS) and Struct of Arrays (SoA).

Array of Structs (AoS)

In AoS, data for a single entity is grouped together in memory. For example, an array of Particle structs where each element contains pos, vel, and accel.

Memory Layout:

p0: [pos, vel, accel]
p1: [pos, vel, accel]

Struct of Arrays (SoA)

In SoA, each field of the struct is stored in its own separate array. This allows the GPU to perform "coherent" or "chunked" memory loads, as multiple values of the same type (e.g., all positions) are stored contiguously.

Memory Layout:

pos0, pos1, ...
vel0, vel1, ...
accel0, accel1, ...

Performance Impact

While SoA is more complex to implement and manage, it significantly improves memory throughput on the GPU. For large datasets (e.g., 50,000 particles), switching from AoS to SoA can result in performance gains of up to 5x because the GPU can load data more efficiently.

An Accelerator represents a specific hardware or software GPU (CPU, CUDA, or OpenCL). Every ILGPU program requires at least one accelerator.

Accelerator Types

| Type | Requirements | Best Use Case | | :--- | :--- | : | | CPUAccelerator | None | Debugging and fallback. Allows use of C# debugging features. | | CudaAccelerator | NVIDIA GPU (GTX 680+) | High-performance NVIDIA hardware. | | CLAccelerator | OpenCL 2.0+ capable GPU | AMD or Intel GPUs. |

Creating Accelerators

You can create specific accelerators using the Context methods. The integer parameter denotes the index of the device to use in multi-device systems.

• CPU: context.CreateCPUAccelerator(0); (Requires using ILGPU.CPU;)
• CUDA: context.CreateCudaAccelerator(0); (Requires using ILGPU.Cuda;)
• OpenCL: context.CreateCLAccelerator(0); (Requires using ILGPU.OpenCL;)

Lifecycle and Disposal

Accelerators must be disposed of in the reverse order of their creation. For example, if you create a Context and then an Accelerator, you must dispose of the Accelerator before the Context is disposed. Device instances do not require disposal.

Implicitly grouped kernels provide a high-level programming model where ILGPU manages the thread grid and group sizes automatically.

Key Characteristics:

• Abstraction: The details of kernel invocation (group sizes, thread participation) are hidden from the user.
• Limitations: Because group sizes are managed by ILGPU, you must not use shared memory, group functionality, or warp-specific intrinsics in these kernels.
• Usage: Use the first parameter as an index type (Index1D, Index2D, or Index3D) to access the global index of the thread.

class ...
{
    static void ImplicitlyGrouped_Kernel(
        [Index1D|Index2D|Index3D] index, 
        [Kernel Parameters]...)
    {
        // Use the index parameter to access the global index of i-th thread in the global thread grid
    }
}

class ...
{
    static void ImplicitlyGrouped_Kernel(
        [Index1D|Index2D|Index3D] index, 
        [Kernel Parameters]...)
    {
        // Kernel code

        // Use the index parameter to access the global index of i-th thread in the global thread grid
    }
}

MemoryBuffer represents an allocated memory region (an array) of a specific value type on an accelerator.

Key usage rules:

• Manual Disposal: While MemoryBuffer instances are eventually released by the Garbage Collector or when the parent Accelerator is disposed, you should manually dispose of them using using blocks or .Dispose() to ensure immediate and explicit control over GPU memory.
• Kernel Usage: You cannot pass a MemoryBuffer directly to a kernel. Instead, you must pass an ArrayView that points to the buffer's memory region.
• Dimensions: ILGPU provides built-in support for 1D, 2D, and 3D buffers. nD-buffers can also be managed using custom index types.

// Allocate a memory buffer on the current accelerator device.
using (var buffer = accelerator.Allocate1D<int>(1024))
{
    // Perform operations using views...
}
// Dispose the buffer after performing all operations

Dynamic specialization allows you to define kernels with constant values that are not known until runtime. This enables the compiler to perform optimizations like constant propagation and loop unrolling that are otherwise impossible with standard parameters.

How it works:

1. Use the SpecializedValue<T> type for kernel parameters.
2. The kernel is precompiled during loading.
3. A final compilation step occurs during the first call with a new, non-cached SpecializedValue<T> combination.
4. Subsequent calls with the same value use the cached specialized instance.

Requirement: Values passed via SpecializedValue<T> must implement the IEquatable interface to ensure correct caching.

Example:

// Standard kernel: 'c' is a runtime variable
static void GenericKernel(ArrayView<int> data, int c)
{
    var globalIndex = Grid.GlobalIndex.X;
    data[globalIndex] = c + 2;
}

// Specialized kernel: 'c' is treated as an inlined constant
static void SpecializedKernel(ArrayView<int> data, SpecializedValue<int> c)
{
    var globalIndex = Grid.GlobalIndex.X;
    data[globalIndex] = c + 2;
}

// Launching
// Generic launch
var genericKernel = accl.LoadStreamKernel<ArrayView<int>, int>(GenericKernel);
genericKernel((<GridDim>, <GroupDim>), buffer.View, 40);

// Specialized launch
var specializedKernel = accl.LoadStreamKernel<ArrayView<int>, SpecializedValue<int>>(SpecializedKernel);
specializedKernel((<GridDim>, <GroupDim>), buffer.View, SpecializedValue.New(40));

class ...
{
    static void GenericKernel(ArrayView<int> data, int c)
    {
        var globalIndex = Grid.GlobalIndex.X;
        // Generates code that loads <i>c</i> and adds the value <i>2</i> at runtime of the kernel
        data[globalIndex] = c + 2;
    }

    static void SpecializedKernel(ArrayView<int> data, SpecializedValue<int> c)
    {
        var globalIndex = Grid.GlobalIndex.X;
        // Generates code that has an inlined constant value
        data[globalIndex] = c + 2; // Will be specialized for every value <i>c</i>
    }

    static void ...(...)
    {
        using var context = new Context();
        using var accl = new CudaAccelerator(context);

        var genericKernel = accl.LoadStreamKernel<ArrayView<int>, int>(GenericKernel);
        ...
        genericKernel((<GridDim>, <GroupDim>), buffer.View, 40);

        var specializedKernel = accl.LoadStreamKernel<ArrayView<int>, SpecializedValue<int>>(GenericKernel);
        ...
        specializedKernel((<GridDim>, <GroupDim>), buffer.View, SpecializedValue.New(40));
        ...
    }
}