lazy-loader

repository·main·Indexed 42 days ago

https://github.com/scientific-python/lazy-loader

A utility to improve Python import times by deferring the loading of subpackages, functions, and external libraries until they are accessed. It provides tools like `attach()` for package __init__.py files, `attach_stub()` for static type checker support, and `load()` for creating lazy module proxies with optional PEP-508 version requirements.

Tokens
1.9K
Snippets
9
Records
12
Agent score
3%

What's inside lazy-loader

  1. Lazily load subpackages in a package __init__.py

    main

    To make subpackages visible to users without incurring import costs, use lazy.attach in your package's __init__.py. This sets up __getattr__ and __dir__ so that subpackages are only loaded when accessed.

    subpackages = [
        ...,
        'filters',
        ...
    ]
    
    import lazy_loader as lazy
    __getattr__, __dir__, _ = lazy.attach(__name__, subpackages)
  2. How lazy loading and shadowing work

    main

    When a function has the same name as a submodule it resides in (e.g., a max_tree function in max_tree.py), the Python import machinery can accidentally overwrite the function attribute with the submodule attribute during import.

    lazy_loader handles this using a _ShadowGuardModule class. This mechanism tracks protected names in a __lazy_shadowed__ attribute and suppresses dictionary updates that would cause a function to be shadowed by its parent module.

  3. Disable lazy loading for debugging with EAGER_IMPORT

    main

    To catch missing imports immediately during development or testing (rather than waiting for attribute access), set the EAGER_IMPORT environment variable:

    • EAGER_IMPORT=1 or EAGER_IMPORT=true: Disables lazy loading (eager mode).
    • EAGER_IMPORT=0 or EAGER_IMPORT="": Re-enables lazy loading.
  4. Load external libraries with version requirements

    main

    You can specify version requirements for external libraries using the require parameter in lazy.load. If the installed version does not meet the requirement, an error will be raised upon attribute access.

    Note: The requirement must use the distribution name (e.g., pyyaml) rather than the import name (e.g., yaml).

    import lazy_loader as lazy
    np = lazy.load("numpy", require="numpy >=1.24")
  5. Enable type checking for lazy imports with attach_stub

    main

    Since static type checkers cannot infer types from lazy imports, use lazy.attach_stub to load type stubs (.pyi files). Note that when using this method, the .pyi files are required for both type checking and runtime execution.

    import lazy_loader as lazy
    __getattr__, __dir__, _ = lazy.attach_stub(__name__, "subpackages.pyi")
  6. Lazily import external libraries with lazy.load

    main

    Use lazy.load to defer the import of external dependencies.

    Important: Avoid loading subpackages directly (e.g., lazy.load('scipy.linalg')), as this will cause the parent package (scipy) to be imported immediately. Instead, load the base package and access the subpackage via attributes.

    Immediate Error Checking: Use error_on_import=True to raise an ImportError as soon as lazy.load is called, rather than waiting for the module to be accessed.

    import lazy_loader as lazy
    
    # Standard lazy load
    sp = lazy.load('scipy')
    sp.linalg.norm(...)
    
    # Load with immediate error checking
    linalg = lazy.load('scipy.linalg', error_on_import=True)
  7. Lazily load subpackages and specific functions

    main

    You can use lazy.attach to lazily load both submodules and specific attributes (functions/classes) within those submodules. This is useful for exposing a flat API while keeping imports deferred.

    from ..util import lazy
    
    __getattr__, __dir__, __all__ = lazy.attach(
        __name__,
        submodules=['rank'],
        submod_attrs={
            '_gaussian': ['gaussian', 'difference_of_gaussians'],
            'edges': ['sobel', 'scharr', 'prewitt', 'roberts',
                      'laplace', 'farid']
        }
    )
  8. Lazily load submodules and attributes with attach()

    main

    Use attach() to replace a package's __getattr__, __dir__, and __all__ so that submodules and specific attributes are only imported when they are first accessed. This is the recommended way for package maintainers to implement lazy loading for their subpackages.

    To use it, call attach() within your package's __init__.py and assign its return values to the package's magic methods.

    Parameters:

    • package_name (str): Typically __name__.
    • submodules (set): A list/set of submodules to attach.
    • submod_attrs (dict): A dictionary mapping submodule names to a list of attributes or functions within that submodule that should be lazily imported.

    Returns:

    • A tuple containing (__getattr__, __dir__, __all__) to be assigned to the package.
    __getattr__, __dir__, __all__ = attach(
        __name__, 
        ["mysubmodule", "anothersubmodule"], 
        {"foo": ["someattr"]}
    )
  9. Support type checkers with attach_stub()

    main

    If you are using lazy loading via attach(), static type checkers might not be able to find your imports. attach_stub() solves this by parsing a .pyi stub file to automatically infer which submodules and attributes should be lazily loaded.

    Parameters:

    • package_name (str): Typically __name__.
    • filename (str): Path to the .py file that has an adjacent .pyi stub file. Typically __file__.

    Returns:

    • A tuple containing (__getattr__, __dir__, __all__) identical to the output of attach().
    __getattr__, __dir__, __all__ = attach_stub(__name__, __file__)
  10. Create a lazy module proxy with load()

    main

    Use load() to return a proxy module that delays the actual import of a dependency until its first attribute is requested. This is useful for avoiding heavy imports (like numpy) at the top level of your module if they are only needed inside specific functions.

    Warning: While you can use load() for subpackages, it causes the parent package to be eagerly loaded. For subpackages, it is better to use attach() in the parent package's implementation.

    Parameters:

    • fullname (str): The full name of the module to import (e.g., 'numpy').
    • require (str, optional): A PEP-508 dependency requirement (e.g., 'numpy >=1.24'). The proxy will raise an error if the requirement is not met.
    • error_on_import (bool): If True, import errors are raised immediately when load() is called. If False (default), errors are raised when the module is first accessed.
    • suppress_warning (bool): If True, prevents a RuntimeWarning when loading subpackages.

    Returns:

    • An importlib.util._LazyModule proxy object.
    import lazy_loader as lazy
    
    # Instead of: import numpy as np
    np = lazy.load("numpy")
    
    def myfunc():
        # numpy is only imported here when np.norm is called
        return np.norm([1, 2, 3])