modshim

Core Mechanism

modshim extends Python's import system by installing a custom ModShimFinder into sys.meta_path. It creates virtual merged modules by performing AST (Abstract Syntax Tree) rewriting on the source code during load time.

Key Concepts

• Mapping: shim() maps three names: lower (original), upper (enhancement), and mount (the new combined entry point).
• AST Rewriting: modshim transforms import X and from X import Y statements to point to the mount point. It also rewrites attribute access (e.g., urllib.response) to ensure consistency.
• Working Module: To prevent circular references when an enhancement subclasses an original component, modshim creates a temporary _working_<module> to hold the original namespace.
• Isolation: Unlike monkey-patching, the original module is never modified. The enhancements only exist under the mount point, preventing global namespace pollution.

modshim allows you to overlay custom functionality onto existing Python modules without modifying their source code. It creates a new, "shimmed" module that combines the original code with your enhancements. This avoids the need for forking, vendoring, or monkey-patching.

To use it, you follow a three-step pattern:

1. Define Enhancements: Create a module that mirrors the structure of the original module, redefining only the parts you want to modify (e.g., by subclassing original classes).
2. Shim the Module: Use the shim() function to merge the original module (lower) with your enhancement module (upper) into a new module name (mount).
3. Import the Result: Import from the new mount name to use the enhanced functionality.

# 1. Define enhancements in 'prefixed_textwrap.py'
from textwrap import TextWrapper as OriginalTextWrapper

class TextWrapper(OriginalTextWrapper):
    def __init__(self, *args, prefix: str = "", **kwargs) -> None:
        self.prefix = prefix
        super().__init__(*args, **kwargs)

# 2. Apply the shim
from modshim import shim
shim(lower="textwrap", upper="prefixed_textwrap", mount="super_textwrap")

# 3. Use the enhanced module
from super_textwrap import wrap
print(wrap("hello", prefix="> "))

You can create a package that automatically applies a shim upon import. This makes your enhancements available to users simply by importing your package.

To do this, call shim() within your package's code. If you omit the upper and mount arguments, they default to the name of the module where shim() is called.

Example super_textwrap.py implementation:

from textwrap import TextWrapper as OriginalTextWrapper
from modshim import shim

class TextWrapper(OriginalTextWrapper):
    # ... enhancement logic ...
    pass

# Apply the shim at import time.
# Defaults: upper=current module, mount=upper
shim(lower="textwrap")

# super_textwrap.py
from textwrap import TextWrapper as OriginalTextWrapper
from modshim import shim

class TextWrapper(OriginalTextWrapper):
    """Enhanced TextWrapper that adds a prefix to each line."""

    def __init__(self, *args, prefix: str = "", **kwargs) -> None:
        self.prefix = prefix
        super().__init__(*args, **kwargs)

    def wrap(self, text: str) -> list[str]:
        original_lines = super().wrap(text)
        if not self.prefix:
            return original_lines
        return [f"{self.prefix}{line}" for line in original_lines]

# Apply the shim at import time. This replaces the 'super_textwrap' 
# module in sys.modules with the new, combined module.
shim(lower="textwrap")

To extend an existing library, create an enhancement package that mirrors the structure of the original library. Use modshim.shim() in your package's __init__.py to overlay your enhancement onto the original module.

1. Mirror the structure: If you want to extend requests.sessions.Session, create your_package/sessions.py.
2. Subclass the original: In your submodule, import the original class (e.g., from requests.sessions import Session as OriginalSession) and subclass it.
3. Initialize the shim: In your_package/__init__.py, call shim(lower="original_package_name"). This defaults the mount point to your package name.

This approach allows you to upgrade top-level functions (like requests.get()) automatically because modshim rewrites their internal references to use your enhanced classes.

# your_package/__init__.py
from modshim import shim

# Overlays 'your_package' on the original 'requests' package.
# Submodules like 'your_package.sessions' will merge with 'requests.sessions'.
shim(lower="requests")

To allow existing code to benefit from enhancements without changing any import statements, you can set the mount parameter to the same name as the lower module. This effectively replaces the original module in sys.modules.

Use Case: Transparent Bug Fixes This is ideal for applying security patches or bug fixes to libraries used throughout your codebase or by third-party dependencies. By applying the shim at application startup (e.g., in __init__.py), all subsequent imports of that module—including those within third-party libraries—will use your patched version.

Caution: Mounting over the original module affects all subsequent imports in your application. Ensure your enhancements are fully backward-compatible with the original module's API.

# my_app/setup_enhancements.py
from modshim import shim

# This replaces 'textwrap' globally for the application
shim(
    lower="textwrap",
    upper="prefixed_textwrap",
    mount="textwrap",
)

# my_app/main.py
from my_app import setup_enhancements  # Apply the shim first
from textwrap import wrap             # Now uses the enhanced version