DOCSEARCHbyAlgolia
Submit Docs

cuTile Python Documentation

repository·main·Indexed 24 days ago

https://github.com/nvidia/cutile-python

A programming language and framework for NVIDIA GPUs designed for writing high-performance kernels using a tile-based programming model. It leverages Tile IR to generate efficient GPU code and includes the cuda.lang library for compiling Python code into cubin (CUDA binary) using a SIMT model. Supports AOT and JIT compilation, and integrates with CuPy and PyTorch via DLPack and the CUDA Array Interface. Compatible with Blackwell and Ampere/Ada GPUs.

Tokens
33.3K
Snippets
77
Records
223
Agent score
85%

What's inside cuTile Python

  1. Overview of cuTile Code Samples

    main
    The samples/ directory contains examples for implementing high-performance GPU kernels in Python using cuTile. cuTile provides a Pythonic interface for CUDA programming concepts such as tiling, shared memory, and warp-level operations. Each sample demonstrates a fundamental GPU operation implemented via a cuTile kernel.

  3. Overview of (Experimental) cuda.lang

    main
    The cuda.lang package (experimental) provides low-level CUDA programming capabilities directly within Python. It exposes the CUDA C++ execution model while providing Pythonic APIs to access modern hardware features like Tensor Cores and the Tensor Memory Accelerator (TMA). It is designed to work alongside cuTile, allowing developers to combine explicit SIMT (Single Instruction, Multiple Threads) control with high-level tile programming through a unified language design and compiler infrastructure.

  4. Work with Global Arrays

    main

    A global array is a container of elements with a specific dtype arranged in a multidimensional space.

    Key Properties:

    • Shape: A tuple of int32 integers representing the length of each dimension. You can access this via Array.shape.
    • Strided Layout: Arrays use a strided memory layout to map logical indices to physical memory.
    • Allocation: New arrays must be allocated on the host and passed to the tile kernel as arguments. The kernel itself can only create new views of existing arrays (e.g., using Array.slice).
    • Memory Constraint: If passing multiple array arguments to a kernel, their memory regions must not overlap; otherwise, behavior is undefined.

    Note on Shape Limits: Array.shape returns int32 values. This caps the maximum representable shape at 2,147,483,647 elements per dimension.

  5. Understand the cuTile Execution Model

    main

    cuTile uses a hierarchical execution model based on a grid of logical blocks:

    • Tile Kernel: Executed by a logical grid of |blocks| (1D, 2D, or 3D).
    • Block: The unit of execution. A block runs on a subset of the GPU.
      • Scalar operations run serially on a single thread within the block.
      • Array operations run collectively in parallel across all threads in the block.
    • Parallelism Model: Tile programs express block-level parallelism only. You cannot access or manage individual threads within a block.
    • Synchronization: Explicit synchronization or communication is not permitted within a block, but is allowed between different blocks.
    • Data vs. Execution: A block is the unit of execution, while a tile is the unit of data. A single block can operate on multiple tiles of different shapes from different global arrays.

  6. Understand the cuTile Data Model

    main

    cuTile uses an array-based programming model where the fundamental data structure is a multidimensional array of a single homogeneous type. Unlike standard Python, cuTile Python exposes only arrays and does not use pointers. This model ensures bounds checking for safety and allows efficient hardware-level load/store operations.

    Key characteristics:

    • Arrays: Containers of elements in a logical multidimensional space.
    • No Pointers: Only arrays are exposed to the user.
    • Memory Safety: Arrays know their bounds, allowing for correctness checks.
    • Interoperability: Any object implementing the DLPack interface or the CUDA Array Interface (e.g., CuPy arrays or PyTorch tensors) can be passed as a kernel argument.

  8. Constraints of the Tile Code Python Subset

    main

    When writing code in the Tile code execution space, you are using a restricted subset of Python. There is no Python runtime within tile code.

    Object Model & Lifetimes

    • Immutability: All objects created within tile code are immutable. Operations that would modify an object instead return a new object. Attributes cannot be added dynamically.
    • Global Arrays: Global arrays are immutable views that allow reading and writing to global device memory.
    • Kernel Safety: When calling a kernel, the caller must ensure:
      • No arrays passed to the kernel alias one another.
      • All arrays remain valid until the kernel execution completes.

    Control Flow

    Python control flow (if, for, while) is supported and can be nested. However, there are specific limitations:

    • Range Steps: The step in a range must be strictly positive. Negative-step ranges like range(10, 0, -1) are not supported and may cause undefined behavior if passed via a variable.

  9. Inter-Kernel Interoperability with SIMT

    main

    cuTile supports inter-kernel interoperability, which refers to operations that do not cross the kernel boundary (i.e., they do not mix tile and SIMT code within a single kernel).

    Supported inter-kernel interoperability tasks include:

    • Writing both tile and SIMT kernels within the same source file.
    • Linking tile and SIMT kernels into a single binary.
    • Passing identical array types to both tile and SIMT kernels.

    Note: Intra-kernel interoperability (mixing tile and SIMT code inside a single kernel) is not currently supported but is planned for the future.

  10. Use Tiles and Scalars in Tile Code

    main

    A tile is an immutable multidimensional collection of elements with a specific dtype.

    Usage Rules:

    • Compile-time Requirements: A tile's shape must be known at compile time, and each dimension must be a power of 2.
    • Scope: Tiles can only be used within tile code, not in host Python code.
    • Scalars: A zero-dimensional tile is a scalar. Numeric literals (e.g., 7 or 3.14) are treated as constant scalars. Unlike Python int/float, scalars have dtype and shape attributes.
    • Lifecycle:
      • Creation: Tiles are created by loading from global arrays (using cuda.tile.load or cuda.tile.gather) or via factory functions like cuda.tile.zeros.
      • Storage: Tiles are stored back into global arrays using cuda.tile.store or cuda.tile.scatter.

    Example of scalar attributes in tile code:

    a = 0
# In tile code, this works:
a.dtype

  11. Understand the cuTile memory model

    main

    cuTile's memory model allows the compiler and hardware to reorder operations to optimize performance. Because of this, memory access ordering across different blocks is not guaranteed unless explicit synchronization is used.

    To coordinate memory accesses between blocks, cuTile uses two attributes for atomic operations:

    1. Memory Order: Defines the ordering semantics of an atomic operation.
    2. Memory Scope: Defines the set of blocks that participate in the ordering.

    Synchronization in cuTile is performed at a per-element granularity, meaning each element in an array participates in the memory model independently.

  12. Construct a KernelSignature for AOT compilation

    main

    To use export_kernel, you must provide a KernelSignature. There are two ways to create one:

    1. Explicit Construction (Recommended): Manually instantiate a KernelSignature object and provide a list of ParameterConstraint objects. This is the safest method as it allows you to precisely define the assumptions (constraints) for each parameter.
    2. Inferred Construction (Prototyping only): Use KernelSignature.from_kernel_args(kernel_func, *example_args). This derives a signature based on the provided example arguments.

    Warning: Using from_kernel_args can lead to undefined behavior if the example arguments satisfy certain properties (like memory alignment) that the kernel then assumes will always be true. Use this only for testing or prototyping.