emsec/hal

The pybind11 benchmark compares compilation time and module size against Boost.Python using a synthetic workload. The workload consists of a growing number of dummy classes (from 1 to 2048) where each class contains four methods with randomly generated signatures.

Key findings from the benchmark:

• Compilation Time: pybind11 is generally faster due to including fewer headers. For a large file with 2048 classes and 8192 methods, pybind11 was approximately 1.2x faster (19.8s vs 116.35s).
• Module Size: pybind11 produces significantly smaller binaries. For the largest test case, the Boost.Python binary was 2.17x larger (16.8 MiB) than the pybind11 output.

    PYBIND11_MODULE(example, m) {
        py::class_<cl034>(m, "cl034")
            .def("fn_000", &cl034::fn_000)
            .def("fn_001", &cl034::fn_001)
            .def("fn_002", &cl034::fn_002)
            .def("fn_003", &cl034::fn_003)
    }

When calling an overloaded function from Python, pybind11 uses a two-pass resolution strategy to determine the correct overload:

1. First Pass (No Conversion): pybind11 attempts to call each overload without allowing any argument conversion (similar to using py::arg().noconvert()).
2. Second Pass (With Conversion): If no overload matches in the first pass, pybind11 attempts a second pass where argument conversion is allowed (unless an overload is explicitly marked with .noconvert()).

Key Rules:

• Priority: pybind11 prefers overloads that require no conversion over those that require conversion.
• Registration Order: Within each pass, overloads are tried in the order they were registered.
• Prepend Tag: You can use the py::prepend() tag during registration to place a specific function at the beginning of the overload sequence, allowing it to be checked before built-in functions.
• No Pattern Prioritization: pybind11 does not prioritize based on the number of arguments or the complexity of the pattern; it only distinguishes between 'no conversion' and 'conversion required'.

If both passes fail, a TypeError is raised.

To avoid expensive copies and allow functions to modify NumPy arrays in-place, use Eigen::Ref<MatrixType> in your C++ function signatures.

How it works:

• Eigen::Ref<const MatrixType>: pybind11 attempts to use Eigen::Map to point directly to the NumPy data. This requires matching data types (e.g., double vs float64) and compatible storage layouts. If incompatible, pybind11 falls back to making a temporary copy.
• Eigen::Ref<MatrixType> (non-const): pybind11 only allows the call if the NumPy array is writeable (a.flags.writeable == True). Modifications made in C++ will be reflected directly in the NumPy array.

Limitations:

• Passing by reference is not supported for sparse types (they are always copied).
• You must ensure storage order compatibility (see Storage orders).

void scale_by_2(Eigen::Ref<Eigen::VectorXd> v) {
    v *= 2;
}

When binding C++ functions to Python, you must specify how memory and object lifetimes are managed to prevent crashes (like double-freeing static data) or resource leaks. pybind11 uses return_value_policy annotations passed to module_::def or class_::def to resolve ownership ambiguity.

Key Concepts:

• Ownership: Determines whether Python's garbage collector should call the C++ destructor when the Python wrapper is deleted.
• Default Policy: py::return_value_policy::automatic. For pointers, this defaults to take_ownership. For other types, it uses move or copy.
• Existing Instances: Policies only apply to new instances. If pybind11 already has a Python wrapper for a specific memory address, it returns the existing wrapper instead of applying the policy to a new one.
• Smart Pointers: If your C++ functions return smart pointers (e.g., std::shared_ptr), you generally do not need to specify a return value policy, as the smart pointer handles lifetime tracking automatically.

/* Example: Preventing a crash when returning a pointer to static data */
Data *get_data() { return _data; }

// WRONG: Default policy assumes ownership and will try to delete static _data
m.def("get_data", &get_data);

// CORRECT: Use reference policy to avoid taking ownership
m.def("get_data", &get_data, py::return_value_policy::reference);

When binding C++ and Python, you have three primary strategies for handling data types across the language boundary:

1. Native C++ types with bindings: Use a native C++ type and wrap it using pybind11-generated bindings. This allows Python to interact with the C++ object directly.
2. Native Python types with wrapping: Use native Python types and wrap them so that C++ functions can interact with them.
3. Type conversions: Use a native C++ type on the C++ side and a native Python type on the Python side. This is often the most "natural" approach because it allows both languages to use their own non-wrapped, native types.

Note on Type Conversions: While type conversions are highly ergonomic, they require a copy of the data to be made during every Python $\leftrightarrow$ C++ transition because the memory layouts of the C++ and Python versions of the same type typically differ.