pyo3-stub-gen

repository·main·Indexed 42 days ago

https://github.com/jij-inc/pyo3-stub-gen

A tool for generating Python stub files (*.pyi) for PyO3-based projects. It provides type hints for Rust-defined Python modules using procedural macros like #[gen_stub_pyfunction] and #[gen_stub_pyclass], bridging the gap between Rust and Python type systems. Supports Python 3.12+ 'type' statement syntax, function overloads, and automatic __init__.py generation for mixed layout projects.

Tokens
39.2K
Snippets
125
Records
162
Agent score
53%

What's inside pyo3-stub-gen

  1. How class method overloads work

    main

    Class methods can be overloaded using the submit! macro combined with gen_methods_from_python!. This allows you to manually define method signatures (including @overload variants) for a #[gen_stub_pyclass].

    Rules for class method overloads:

    • Multiple methods with the same name must use the @overload decorator in the Python string.
    • At most one method can have is_overload = false (this represents the actual implementation).
    • The @overload decorator is automatically detected from the provided Python stub syntax.
    use pyo3::prelude::*;
    use pyo3_stub_gen::{derive::*, inventory::submit};
    
    #[gen_stub_pyclass]
    #[pyclass]
    pub struct ManualSubmit {}
    
    submit! {
        gen_methods_from_python! {
            r"#
            class ManualSubmit:
                @overload
                def echo(self, obj: int) -> int: ...
    
                @overload
                def echo(self, obj: float) -> float: ...
            "#
        }
    }
    
    #[pymethods]
    impl ManualSubmit {
        fn echo<'arg>(&self, obj: Bound<'arg, PyAny>) -> Bound<'arg, PyAny> {
            obj
        }
    }
  2. How submodules are detected and imported in stubs

    main

    In a Mixed Python/Rust layout, pyo3-stub-gen automatically detects module hierarchies based on the module paths provided in #[gen_stub_pyfunction(module = "...")].

    Even if you only explicitly define leaf modules, the generator will:

    1. Parse the dot-separated module names to identify parent-child relationships.
    2. Create necessary intermediate parent modules.
    3. Automatically add relative import statements to the parent's __init__.pyi file so submodules are accessible.

    Example Import Generation: If you have modules mixed.main_mod.mod_a and mixed.main_mod.mod_b, the generated mixed/main_mod/__init__.pyi will contain:

    from . import mod_a
    from . import mod_b
  3. Reference Rust types in Python stub syntax using RustType

    main

    You can reference Rust types directly within Python stub strings using the pyo3_stub_gen.RustType["TypeName"] marker. This leverages the PyStubType trait to map Rust types to their appropriate Python equivalents.

    Common mappings:

    • RustType["Vec<i32>"] $\rightarrow$ typing.Sequence[int]
    • RustType["i32"] $\rightarrow$ int
    submit! {
        gen_function_from_python! {
            r#"
            def sum_list(values: pyo3_stub_gen.RustType["Vec<i32>"]) -> pyo3_stub_gen.RustType["i32"]:
                """Sum a list of integers"""
            "#
        }
    }
  4. How default values are handled for different member types

    main

    The pyo3-stub-gen tool handles default values differently based on how the member is defined in Rust:

    1. Class Attributes (#[classattr])

    For constants defined within a #[pymethods] block using #[classattr], the default value is automatically extracted from the constant value. The #[gen_stub(default = ...)] attribute is optional here.

    Rust:

    #[gen_stub_pymethods]
    #[pymethods]
    impl MyClass {
        #[classattr]
        const VERSION: &'static str = "1.0.0";
    }

    Stub:

    class MyClass:
        VERSION: str = '1.0.0'

    2. Getters (#[pyo3(get)] or custom methods)

    For properties, the default value is embedded into the docstring of the @property method using a markdown-style Python block.

    Rust:

    #[gen_stub_pyclass]
    #[pyclass]
    struct MyClass {
        #[pyo3(get, set)]
        #[gen_stub(default = MyClass::default().value)]
        value: i32,
    }

    Stub:

    class MyClass:
        @property
        def value(self) -> int:
            r"""
            ```python
            default = 42
            ```
            """

    3. Setters (#[pyo3(set)])

    When a member has both a getter and a setter, the default value is included in the docstrings of both the @property and the @value.setter methods.

  5. Compare TypeAlias vs type statement syntax

    main

    The output format of generated type aliases depends on your pyproject.toml configuration:

    With use-type-statement = true (Python 3.12+):

    type SimpleAlias = int | None
    type StrIntMap = dict[str, int]

    With default settings (Pre-3.12):

    from typing import TypeAlias
    SimpleAlias: TypeAlias = int | None
    StrIntMap: TypeAlias = dict[str, int]
    # Generated with use-type-statement = true
    type SimpleAlias = int | None
    type StrIntMap = dict[str, int]
    
    # Generated with default settings
    from typing import TypeAlias
    SimpleAlias: TypeAlias = int | None
    StrIntMap: TypeAlias = dict[str, int]
  6. Understand limitations of Python stub syntax

    main

    Be aware of the following current limitations when using Python stub syntax:

    • No Class-Level Python Stubs: gen_stub_pyclass(python = "...") is not currently supported. To define class methods using Python syntax, use gen_methods_from_python! instead.
    • No Compile-Time Type Validation: The tool does not validate that your Python type annotations match your Rust implementation at compile-time. Validation only occurs when you run static analysis tools like mypy or pyright on the generated .pyi files.
    • Limited AST Support: Extremely complex Python type patterns might not be supported. If a pattern fails, fallback to typing.Any or use manual overrides.
  7. Mix automatic and manual method submission

    main

    For classes where most methods can be automatically generated but a few require complex type annotations (like Callable) or overloads, you can combine #[gen_stub_pymethods] with manual submit! blocks.

    Important Rules:

    1. Use #[gen_stub(skip)] on Rust methods that you want to handle manually.
    2. Place submit! blocks before the #[gen_stub_pymethods] implementation block. This ensures that pyo3-stub-gen orders @overload definitions correctly based on their source position in the Python snippet.
    #[gen_stub_pyclass]
    #[pyclass]
    pub struct PartialManualSubmit {}
    
    // 1. Submit manual entries (overloads/complex types) FIRST
    submit! {
        gen_methods_from_python! {
            r#"
            class PartialManualSubmit:
                @overload
                def echo_overloaded(self, obj: int) -> int: ...
            "#
        }
    }
    
    // 2. Then use the proc-macro for the rest
    #[gen_stub_pymethods]
    #[pymethods]
    impl PartialManualSubmit {
        #[gen_stub(skip)]
        pub fn echo_overloaded<'arg>(&self, obj: Bound<'arg, PyAny>) -> PyResult<Bound<'arg, PyAny>> {
            // ...
        }
    }
  8. Choose between Pure Rust and Mixed Python/Rust layouts

    main

    The way pyo3-stub-gen generates stub files depends on your project layout. Choosing the correct layout is critical for ensuring that submodules are correctly represented and packaged by maturin.

    Pure Rust Layout

    Use this when your project is a single module with no submodules and no existing Python source files.

    • Configuration: Omit python-source from [tool.maturin] in pyproject.toml.
    • Stub Generation: Generates a single {package_name}.pyi file at the project root.
    • Limitation: maturin will only package this single file. Directory-based stubs are ignored in this layout.

    Use this if you have Python source files OR if you use PyO3's add_submodule to create a module hierarchy.

    • Configuration: Add python-source = "<path>" (e.g., python-source = "python") to [tool.maturin] in pyproject.toml.
    • Stub Generation: Generates __init__.pyi files within the directory structure corresponding to your modules (e.g., {python_root}/{module_path}/__init__.pyi).
    • Benefit: This is the only way to ensure submodules are properly packaged and recognized by type checkers.
    # Example Mixed Layout Configuration
    [tool.maturin]
    python-source = "python"
    module-name = "mypackage.main_mod"
  9. How pyo3-stub-gen handles link resolution

    main

    The documentation generator uses Haddock-style rules to resolve links for items. This ensures that re-exported items point to their original public module rather than the module they were re-exported from.

    1. If exported from current module: Link to the current module.
    2. Else if exported from a public module: Link to that public module.
    3. Else (private module only): No link is created.

    Private Module Detection

    A module is considered private if any part of its path starts with an underscore (_). For example:

    • package._internal is private.
    • package.public is public.
  10. How default values for function arguments are implemented

    main

    The library supports two distinct ways to specify default values for function arguments, which are then converted into the final Python stub syntax (param_name: Type = default_value).

    1. Rust-based Approach (PyO3 Signature)

    In this approach, default values are extracted from the PyO3 #[pyo3(signature = ...)] attribute in your Rust code.

    How it works:

    • Signature Parsing: The generator parses the PyO3 signature, identifying positional-only (/), keyword-only (*), and standard parameters. When it encounters an assignment (e.g., param = value), it captures the Rust expression.
    • Type-safe Conversion (RustType): If the parameter has a known Rust type, the generator uses compile-time type checking (let v: #r#type = #expr) and a runtime utility fmt_py_obj to convert the Rust value into its proper Python string representation.
    • Direct String Conversion (OverrideType): If a type override is used, the generator performs direct string conversion, including translating Rust boolean literals (true/false) to Python literals (True/False).

    2. Python-based Approach (Direct Stub Syntax)

    In this approach, default values are extracted directly from a Python stub string provided to the generator.

    How it works:

    • AST Parsing: The generator uses rustpython-parser to parse the Python stub string into an Abstract Syntax Tree (AST).
    • AST to String Conversion: The generator traverses the AST to convert Python expressions back into valid Python syntax strings. This includes handling:
      • Literals: None, True, False, integers, floats, and strings.
      • Attribute access: e.g., MyEnum.VARIANT.
      • Unary operations: e.g., -42.
      • Collections: Recursive conversion of list, tuple, and dict.
    • Direct Usage: Because the input is already Python syntax, the generated code simply returns the extracted string directly without further conversion.

    Comparison of Conversion Logic

    FeatureRust-based ApproachPython-based Approach
    Source#[pyo3(signature = ...)]Python stub string
    Conversion Methodfmt_py_obj (Runtime) or String mappingAST traversal to String
    Type SafetyCompile-time checked via Rust typesNo Rust type checking performed
  11. How stub file paths are determined

    main

    The output path for generated stub files is determined strictly by the detected layout type from pyproject.toml:

    Layout TypeOutput Path PatternFormat
    Pure Rust$CARGO_MANIFEST_DIR/{package_name}.pyiSingle file
    Mixed Python/Rust{python-source}/{module_path}/__init__.pyiDirectory-based

    Path Normalization Rules:

    1. Dashes to Underscores: Package names with dashes (e.g., my-package) are converted to underscores (my_package).
    2. Dots to Slashes: Module path separators (e.g., sub.module) are converted to filesystem separators (sub/module).
  12. Understand how type prefixes are stripped for display

    main

    To make documentation more readable, pyo3-stub-gen simplifies type display text by stripping common module prefixes. This applies to standard library types, package-specific types, and internal modules.

    Stripping Rules:

    • Standard Library: Prefixes like typing., builtins., and collections.abc. are removed (e.g., typing.Optional[T] becomes Optional[T]).
    • Package Prefixes: Uses a heuristic (lowercase.Uppercase pattern) to strip package paths (e.g., mypackage.submodule.MyClass becomes MyClass).
    • Internal Modules: Prefixes for internal modules are stripped (e.g., _internal.PrivateClass becomes PrivateClass).