xtensor-python

repository·master·Indexed 18 days ago

https://github.com/xtensor-stack/xtensor-python

Python bindings for the xtensor C++ multi-dimensional array library. It enables high-performance, in-place use of numpy arrays within C++ code, supporting numpy-style broadcasting and STL-compliant APIs. The library provides container types like pyarray for dynamic shape synchronization and pytensor for static, stack-allocated shapes, as well as xt::pyvectorize to create NumPy universal functions (ufuncs) from C++ scalar functions.

Tokens
5.4K
Snippets
19
Records
27
Agent score
63%

What's inside xtensor-python

  1. What is xtensor-python?

    master

    xtensor-python provides Python bindings for the xtensor C++ multi-dimensional array library. It allows for the in-place use of NumPy arrays within C++ code, enabling you to leverage xtensor's features directly on Python data structures.

    Key capabilities include:

    • C++ Universal Functions (ufuncs): Applying C++ scalar functions to NumPy arrays.
    • Broadcasting: Using xtensor's broadcasting rules on NumPy arrays.
    • STL Compliance: Providing APIs that are compliant with the C++ Standard Template Library.
    • Seamless Interoperability: Using Python's Buffer Protocol via pybind11 to access NumPy data structures in C++ without exposing new types to Python.
  2. Explore xtensor-python API Containers

    master

    The xtensor-python API provides specialized container types designed to bridge C++ xtensor arrays with Python's NumPy ecosystem. The primary container types available are:

    • pyarray: Likely used for wrapping or interfacing with NumPy arrays.
    • pytensor: Likely used for interfacing with xtensor tensors within a Python context.

    Refer to the specific documentation for pyarray and pytensor for detailed method signatures and usage patterns.

  3. Understand Fall-back Cast behavior for non-templated functions

    master

    If you cannot use templates (e.g., when overriding non-templated functions), you can still bind your module using xtensor-python. However, this approach involves a data copy from the NumPy array to a temporary xtensor object.

    Important Implications:

    • Performance: It is more costly than using xt::pyarray or xt::pytensor because it requires a copy.
    • Side Effects: If your C++ function accepts a reference (e.g., void foo(xt::xarray<double>& a)), any changes made to the data inside the function will be applied to the temporary copy and will not be reflected in the original NumPy array in Python.

    Comparison of C++ signatures and Python behavior:

    1. void foo(const xt::xarray<double>& a); (Constant reference) $\rightarrow$ Python results in a copy to a temporary variable (read-only).
    2. void foo(xt::xarray<double>& a); (Reference) $\rightarrow$ Python results in a copy to a temporary variable (changes are lost).
    3. void foo(xt::xarray<double> a); (Copy) $\rightarrow$ Python results in a copy to a temporary variable, and then another copy into a.
  4. Restrict C++ template functions to xtensor types using SFINAE

    master

    To create a flexible C++ API that accepts various types but specifically allows xt::pyarray or xt::pytensor from Python, you can use SFINAE (Substitution Failure Is Not An Error).

    This allows you to define a templated C++ function that works with standard xtensor types, while the Python bindings specifically target the xtensor-python wrappers. This ensures that when Python passes a NumPy array, it is treated as an xtensor without unnecessary overhead, while still allowing the C++ side to be type-safe and restricted to xtensor types.

    template <class T>
    void times_dimension(T& t);
  5. How pytensor and pyarray work together

    master

    xtensor-python provides two container types that wrap numpy arrays in-place to provide xtensor semantics. Both support numpy-style APIs.

    • pyarray: Has a dynamic number of dimensions. Like numpy arrays, it can be reshaped with a shape of a different length, and the new shape is reflected on the Python side.
    • pytensor: Has a compile-time number of dimensions specified via a template parameter. Shapes are stack-allocated, making pytensor significantly faster than pyarray for expression evaluation.
  6. Use xt::pyarray to interface with NumPy arrays

    master

    The xt::pyarray class is the core interface for working with NumPy arrays within C++. It allows you to wrap a NumPy array (from Python) into an xtensor object, enabling you to use xtensor's high-performance algorithms and expressions directly on data owned by Python/NumPy. This provides a seamless bridge between Python's memory management and C++'s computational power.

    // Note: The specific API methods are documented via Doxygen in the source.
    // Typically, you wrap a NumPy array like this:
    // xt::pyarray<double> array = ...;
  7. Understand the difference between pyarray and pytensor

    master

    xtensor-python provides two container types that wrap NumPy arrays, serving as Python counterparts to xarray and xtensor. Choosing between them depends on whether you need dynamic shape synchronization or high-performance static shape computation.

    pyarray

    • Behavior: Acts like xarray with a dynamic shape.
    • Synchronization: It does not copy shape or strides; instead, it reads them from the NumPy array every time they are needed.
    • Use Case: Use pyarray if you need changes made to the NumPy array's shape or strides in Python to be immediately reflected in your C++ code.

    pytensor

    • Behavior: Acts like xtensor with a static, stack-allocated shape.
    • Synchronization: The shape of the NumPy array is copied into the pytensor object upon creation. Reshaping the NumPy array in Python will not be reflected in the pytensor object in C++.
    • Use Case: Use pytensor when you want more efficient computation of shapes and broadcasting, and you do not require real-time shape synchronization between Python and C++.
  8. Import the numpy C API in a multi-file extension module

    master

    In projects spanning multiple source files, the FORCE_IMPORT_ARRAY macro must be defined exactly once.

    To manage this correctly:

    • Define #define FORCE_IMPORT_ARRAY in the main file that contains the module initialization code.
    • Ensure this definition occurs before including any xtensor-python headers (even indirectly via other headers).
    • In all other source files, you can include xtensor-python headers normally without defining FORCE_IMPORT_ARRAY.
    // image.hpp
    // Do NOT define FORCE_IMPORT_ARRAY here
    #include "xtensor-python/pyarray.hpp"
    
    class image
    {
    // ....
    private:
        xt::pyarray<double> m_data;
    };
    
    // image.cpp
    // Do NOT define FORCE_IMPORT_ARRAY here
    #include "image.hpp"
    // definition of the image class
    
    // main.cpp
    // FORCE_IMPORT_ARRAY must be define ONCE, BEFORE including
    // any header from xtensor-python (even indirectly)
    #define FORCE_IMPORT_ARRAY
    #include "image.hpp"
    PYBIND11_MODULE(plugin_name, m)
    {
        xt::import_numpy();
        //...
    }
  9. Build the HTML documentation

    master

    To build the documentation, you need doxygen, sphinx, and breathe. You can install breathe via pip or conda.

    1. Install breathe: pip install breathe or conda install -c conda-forge breathe.
    2. Navigate to the docs subdirectory.
    3. Run make html.
    # From the docs subdirectory
    make html
  10. Install xtensor-python from source with CMake

    master

    You can build and install xtensor-python from source using CMake. All installation methods place the cmake project configuration file in the correct location, allowing third-party projects to locate the headers using find_package.

    ### Unix platforms
    ```bash
    mkdir build
    cd build
    cmake -DCMAKE_INSTALL_PREFIX=/path/to/prefix ..
    make install

    Windows platforms

    mkdir build
    cd build
    cmake -G "NMake Makefiles" -DCMAKE_INSTALL_PREFIX=/path/to/prefix ..
    nmake
    nmake install
  11. Import the numpy C API in a single-file extension module

    master

    When creating a self-contained extension module in a single C++ file, you must ensure the numpy C API is correctly imported to avoid linker errors.

    1. Define FORCE_IMPORT_ARRAY before including any xtensor-python headers.
    2. Call xt::import_numpy() within your module initialization function (e.g., inside the PYBIND11_MODULE block).
    #define FORCE_IMPORT_ARRAY
    #include "xtensor-python/pyarray.hpp"
    
    PYBIND11_MODULE(plugin_name, m)
    {
        xt::import_numpy();
        //...
    }