Ziggy Pydust

repository·develop·Indexed 21 days ago

https://github.com/spiraldb/ziggy-pydust

A framework for writing and packaging native Python extension modules in Zig. It features automatic argument wrapping via comptime, a Pytest plugin for running Zig tests, and support for the Python Buffer Protocol and class definitions. Requires Zig 0.14.0 and CPython >=3.11.

Tokens
8.7K
Snippets
31
Records
44
Agent score
73%

What's inside ziggy-pydust

  1. Overview of Ziggy Pydust

    develop

    Ziggy Pydust is a framework designed for writing and packaging native Python extension modules using the Zig programming language. It provides tools for seamless interop between Zig and Python, including:

    • Packaging: Tools to package Zig-based Python extensions.
    • Testing: A Pytest plugin that allows you to discover and run Zig tests within your Python test suite.
    • Comptime Interop: Uses Zig's comptime capabilities to automatically wrap and unwrap arguments, enabling direct interop with native Zig types.
  2. Use Self-managed Mode for complex Zig builds

    develop

    By default, Pydust generates pydust.build.zig (for bootstrapping) and build.zig (based on your pyproject.toml).

    If you need full control over your build logic, enable Self-managed Mode. In this mode, Pydust only generates pydust.build.zig, and you are responsible for managing your own build.zig file.

    To enable this:

    1. Set self_managed = true under [tool.pydust] in pyproject.toml.
    2. Remove all [[tool.pydust.ext_module]] entries from pyproject.toml.
    [tool.pydust]
    self_managed = true
  3. Implement inheritance in Pydust

    develop

    You can define a subclass of a Zig Pydust class by including the parent class struct as a field within the subclass struct.

    Limitations:

    • It is currently not possible to create a subclass of a Python class.

    To invoke methods from the parent class, use py.super(Type, self), which returns a proxy py.PyObject similar to Python's built-in super().

    const SubClass = py.class(struct {
        parent: ParentClass, // Include parent as a field
    
        pub fn some_method(self: *Self) void {
            // Use super to call parent methods
            _ = py.super(ParentClass, self);
        }
    });
  4. Use instance attributes in Pydust

    develop

    Instance attributes allow storing state without custom getters/setters. In Pydust, attributes wrap the type in a struct definition: struct { value: T }.

    Important:

    • In Zig, you must access the attribute via the .value field (e.g., instance.my_attr.value).
    • Attributes are currently read-only.
  5. Use the Python Buffer Protocol for zero-copy operations

    develop

    Python objects that implement the Buffer Protocol can be used with zero-copy in Ziggy Pydust. This allows functions to accept Python array.array objects, NumPy arrays, or any other object implementing the protocol without the overhead of copying data. When working with buffers, ensure you understand request types; common request types are represented in Pydust via py.PyBuffer.Flags (for example, py.PyBuffer.Flags.FULL_RO).

    --8<-- "example/buffers.zig:sum"
    --8<-- "test/test_buffers.py:sum"
  6. How Python modules work in Pydust

    develop

    In Pydust, Python modules are implemented as Zig structs. When a struct is registered as a module, Pydust automatically generates and exports a #!c PyObject *PyInit_<modulename>(void) function, which enables the module to be imported by Python.

    Key characteristics of Pydust modules:

    • Internal State: Unlike standard Python modules, native Pydust modules can carry private internal state via struct fields.
    • Initialization: Fields that require calling into Python (and thus cannot be initialized at comptime) must be initialized within the module's __init__ function.
    • Stateful Functions: Module functions that take a *Self or *const Self argument receive a pointer to the module's internal state.
    • Argument Handling: Arguments are accepted as pointers to const structs. Pydust uses these struct field names to automatically generate Python function docstrings.
    • Submodules: Submodules can be nested, but they are not true Python packages. You can import a submodule using from example.modules import submodule, but you cannot use the from example.modules.submodule import world syntax.
  7. Handle slices and memory safety in Pydust

    develop

    When working with slices (e.g., []const u8) in Pydust functions, observe these memory safety rules:

    1. Returning Slices: You cannot return slices from Pydust functions. Pydust cannot manage the deallocation of these slices once they are copied into Python.
    2. Receiving Slices: You can accept slices as function arguments. However, the underlying bytes are only guaranteed to live for the duration of the function call. If you need the data to persist, you must copy it.
  8. Manage memory in Pydust using incref and decref

    develop

    Pydust does not perform implicit memory management and follows CPython's reference counting semantics. All Pydust Python types (e.g., py.PyObject, py.PyString) provide incref() and decref() member functions, which correspond to the CPython Py_INCREF and Py_DECREF macros.

    Key Rules:

    • Borrowing vs. Stealing: When a function is called from Python, arguments are typically borrowed references. If a Pydust method steals a reference (takes ownership), you must call .incref() on the borrowed object beforehand to ensure the reference count remains valid after the steal.
    • Ownership: Most Pydust functions do not steal references. Functions that do steal references are rare and typically follow a naming convention using fromOwned (e.g., someFunctionFromOwned).
    • Manual Cleanup: If you create a new Python object within Zig (e.g., via py.PyString.fromSlice), you are responsible for calling .decref() on it when it is no longer needed to prevent memory leaks.
  9. Implement Dunder (Magic) methods

    develop

    Dunder methods allow you to override Python built-in operators.

    Key Concepts

    • object: Refers to a pointer to a Pydust type, a py.PyObject(root), or other Pydust types like py.PyString.
    • CallArgs(root): A Zig struct used for args and kwargs. Fields with default values are treated as keyword arguments.

    Shorthand Signatures

    For convenience, you can use these shorthand signatures:

    • binaryfunc: fn(*Self, object) !object
    • unaryfunc: fn(*Self) !object
    • inquiry: fn(*Self) !bool
  10. Initialize and finalize the Python interpreter for Zig tests

    develop
    Because Zig tests are spawned as separate processes, the Python interpreter is not automatically available within the Zig test environment. You must manually manage the interpreter lifecycle within your Zig tests using py.initialize() and defer py.finalize() to ensure the environment is correctly set up and torn down.
  11. Chain string operations using PyString.appendSlice

    develop

    The PyString.appendSlice method is designed for performance and ergonomics when chaining multiple append operations. Unlike many other methods, appendSlice steals a reference to the string it is called on.

    To use it safely in a chain, you must re-assign the result of the call to your variable, as the method returns the updated reference. If you are working with a borrowed reference from Python, you must call .incref() before calling .appendSlice to prevent the reference from being lost during the 'steal'.

    var s = py.PyString.fromSlice("Hello ");
    s = s.appendSlice("1, ");
    s = s.appendSlice("2, ");
    s = s.appendSlice("3");
    return s;