DOCSEARCHbyAlgolia
Submit Docs

mypy-boto3-builder

repository·main·Indexed 20 days ago

https://github.com/youtype/mypy_boto3_builder

A type annotations generator for major AWS Python typing projects, including types-boto3, boto3-stubs, types-aiobotocore, and types-aioboto3. It allows developers to create custom, optimized type annotation packages to improve IDE support and static analysis (mypy, pyright) for specific AWS usage by extracting service definitions from botocore and rendering them as Python type stubs (.pyi files).

Tokens
17.5K
Snippets
60
Records
78
Agent score
71%

What's inside mypy-boto3-builder

  1. How mypy-boto3-builder works

    main

    mypy-boto3-builder is a code generator that creates Python type stubs (.pyi files) for AWS SDK libraries like boto3, aioboto3, and aiobotocore.

    It follows a pipeline architecture:

    1. Parsing: Extracts AWS service definitions from botocore using Parsers.
    2. Structuring: Converts raw definitions into an internal data model using Structures (Data Transfer Objects).
    3. Generating: Transforms the internal model into a target library format using Generators (Strategy Pattern).
    4. Rendering: Uses Jinja2 Templates to produce the final Python code.

    This pipeline allows the tool to support hundreds of AWS services and multiple target SDK formats by simply swapping generators and templates.

  2. How mypy-boto3-builder works

    main

    The mypy-boto3-builder is a fully automated tool that generates type annotations for AWS services. It ensures high-fidelity type safety for boto3 users by performing the following:

    • Full Coverage: Covers all available botocore services.
    • Schema-Driven Annotations: Extracts valid type annotations for every public class and method of every botocore service directly from botocore schemas.
    • Rich Metadata: Includes up-to-date documentation and direct links to documentation for every method.
    • Code Quality: All generated code is processed by ruff to ensure readability and standard formatting.

  3. Quickstart with mypy-boto3-builder

    main

    To generate a custom package of type annotations for your specific AWS needs, follow these steps:

    1. Install uv.
    2. Run the builder using uvx mypy_boto3_builder.
    3. Follow the interactive prompts to answer questions about your requirements.
    4. Once the process completes, you will receive a custom package which you can then install into your environment.
    uvx mypy_boto3_builder

  4. Install type annotations for aiobotocore

    main

    To enable type checking and code completion for aiobotocore, install the types-aiobotocore package.

    Note on Lite versions: The types-aiobotocore-lite package is more RAM-friendly but does not provide session.create_client overloads. If you use the lite version, you must provide explicit type annotations.

    Requirement: Ensure you have a type checker like mypy or pyright installed.

    # Standard installation
python -m pip install 'types-aiobotocore[essential]'

# RAM-friendly Lite version (requires explicit type annotations)
python -m pip install 'types-aiobotocore-lite[essential]'

  6. How to generate type stubs for a target library

    main

    The generation process follows a four-step data flow: Service Discovery, Parsing, Generation, and Template Rendering.

    # 1. Discover available AWS services
service_names = get_available_service_names()

# 2. Parsing Phase
for name in service_names:
    service_model = session.get_service_model(name)
    parser = ServicePackageParser(service_model, name)
    service_package = parser.parse()

    # 3. Generation Phase
    # Use a specific generator (e.g., Boto3Generator, AioBoto3Generator)
    generator = Boto3Generator()
    package = generator.generate_service_package(service_package)

    # 4. Template Rendering
    writer = PackageWriter(output_path)
    writer.write_service_package(package, templates_path)
    # Discover available AWS services
service_names = get_available_service_names()
# Returns: ['s3', 'ec2', 'lambda', ...]

# For each service:
service_model = session.get_service_model(service_name)
parser = ServicePackageParser(service_model, service_name)
service_package = parser.parse()

# Transform to target format:
generator = Boto3Generator()
package = generator.generate_service_package(service_package)

# Render to Python code:
writer = PackageWriter(output_path)
writer.write_service_package(package, templates_path)

  7. Set up the development environment

    main

    To develop on this project, you need to install uv and synchronize the repository dependencies with all extras and development tools enabled.

    1. Install uv using the official installer:
      curl -LsSf https://astral.sh/uv/install.sh | sh
    2. Install all dependencies, including development and extra features, using uv sync:
      uv sync --all-extras --dev
    curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --all-extras --dev

  8. Build type annotations using the CLI

    main

    You can build type annotations using pip and the mypy_boto3_builder module.

    1. Install the specific versions of boto3 and botocore you want to generate types for.
    2. Install mypy-boto3-builder.
    3. Run the builder module specifying the output directory, products, and output type.
    4. Install the resulting .whl files from the output directory.
    # Install preferred version of `boto3`
python -m pip install boto3==1.35.71 botocore==1.35.71

# Install `mypy-boto3-builder`
python -m pip install mypy-boto3-builder

# Build all packages in typings directory
python -m mypy_boto3_builder ./typings --product types-boto3 types-boto3-services --output-type wheel

# Or specify required services explicitly
python -m mypy_boto3_builder ./typings --product types-boto3 types-boto3-services --output-type wheel -s ec2 s3

# Install custom `types-boto3` packages
python -m pip install ./typings/*.whl

  9. Build type annotations using uv (Recommended)

    main

    The recommended way to run the builder is using uvx. This allows you to run the builder and interact with it directly. For better results, you should explicitly set the boto3 version to match your project's requirements using the --with flag.

    # run builder and chat with him :)
uvx mypy_boto3_builder

# set library version explicitly for better results
uvx --with 'boto3==1.35.71' mypy-boto3-builder

  10. Install type annotations for aioboto3

    main

    To enable type checking and code completion for aioboto3, install the types-aioboto3 package.

    Note on Lite versions: The types-aioboto3-lite package is more RAM-friendly but does not provide session.client or session.resource overloads. If you use the lite version, you must provide explicit type annotations.

    Requirement: Ensure you have a type checker like mypy or pyright installed.

    # Standard installation
python -m pip install 'types-aioboto3[essential]'

# RAM-friendly Lite version (requires explicit type annotations)
python -m pip install 'types-aioboto3-lite[essential]'

  11. Verify the build and run pre-commit checks

    main

    Use the provided scripts to ensure the repository is in a working state and to run manual checks before committing code.

    • Build and verify: Run the build script to check if the repository is functioning correctly:
      ./scripts/build.sh
    • Manual pre-commit: Run the pre-commit script manually to validate changes:
      ./scripts/before_commit.sh
    ./scripts/build.sh
./scripts/before_commit.sh

  12. Extend mypy-boto3-builder with new target libraries

    main

    To add support for a new target library (e.g., a new async SDK), follow these steps:

    1. Create a Generator: Inherit from BaseGenerator and implement the library-specific transformation logic.
    2. Create Templates: Add corresponding Jinja2 templates in the templates/ directory.
    3. Register: Add the new generator to the CLI options.