Python Enhancement Proposals (PEPs)

PEP 817 proposes an extension to the Python wheel packaging format called Wheel Variants. This mechanism allows package maintainers to declare multiple build variants for the same package version, enabling installers to automatically select the most appropriate variant based on specific hardware (e.g., GPU acceleration like CUDA or ROCm) or software (e.g., specific dependency ABIs) characteristics.

Key components of the proposal include:

• Wheel Variant format: An evolution of the wheel format that distinguishes wheels by hardware or software attributes.
• Variant provider plugin interface: An interface for installers to dynamically detect platform attributes and select the best-suited wheel.

The goal is to allow standard installation commands (e.g., {tool} install <package>) to automatically resolve to the optimal hardware-specific build without requiring users to manually select package names or separate indexes.

PEP 723 defines a standard for embedding dependency metadata directly within a single Python script file. This is designed for users writing scripts that are not intended for formal distribution (e.g., shared via Gists, email, or internal tools) but still require specific dependencies to run.

Key design decisions include:

• Allowing [tool] tables: Enables script runners to support custom configurations, such as injecting dependency resolution data or instructing scripts to run in specific containers.
• Tool Behavior: Tools are permitted to modify their behavior based on whether the script contains embedded metadata, even when the script is not the sole input (e.g., a linter running on a directory).
• Avoiding pyproject.toml: While pyproject.toml is standard for projects, it is considered too complex for simple, single-file scripts that are often shared informally.
• Avoiding Import Inference: Automatically inferring requirements from import statements is avoided due to parsing difficulties, the inability to map module names to PyPI package names reliably, and security risks (e.g., dependency confusion attacks).
• Avoiding Python Variable Metadata: Using special variables like __dependencies__ is avoided because it would require all consumers to implement a full Python parser that can handle multiple Python versions.

PEP 817 discusses several architectural approaches for handling package variants (e.g., CPU vs GPU versions of libraries like PyTorch).

Provider Plugins Approach

Instead of hardcoding compatibility logic into package managers, this approach uses provider plugins. This allows:

• Independent maintenance of variant properties by those with specific hardware knowledge.
• Frequent updates without requiring package manager updates.
• Decision-making by the package maintainer regarding which provider to use.

Resolver Package Approach (Alternative)

An alternative considered was using a main package as a "resolver" that directs to specific subpackages (e.g., torch directing to torch-cpu or torch-cu129). This was deemed problematic due to:

• Risk of name squatting and users accidentally pinning to old variant projects.
• Complexity in dependency resolution (the resolver would need to query all packages before performing resolution).
• Loss of the one-to-one mapping between dependencies and installed packages.

Docutils is organized into several key modules and packages that handle the document processing lifecycle:

• docutils: Core package containing base classes like Component, SettingsSpec (runtime settings), and TransformSpec (transforms).
• docutils.core: Provides the Publisher facade class and convenience functions.
• docutils.frontend: Manages runtime settings, configuration files, and command-line argument processing.
• docutils.io: Provides a uniform API for low-level input and output.
• docutils.nodes: Contains the document tree element class library and Visitor pattern base classes.
• docutils.statemachine: A finite state machine used for text filters and parsers (e.g., reStructuredText).
• docutils.utils: Utility functions and the Reporter logger class.
• docutils.parsers: Markup parsers. Use get_parser_class(parser_name) to retrieve a parser module. The base class is Parser.
• docutils.readers: Context-aware input readers. Use get_reader_class(reader_name) to retrieve a reader module. The base class is Reader.
• docutils.writers: Output format writers. Use get_writer_class(writer_name) to retrieve a writer module. The base class is Writer.
• docutils.transforms: Tree transformation logic. Transformer manages and applies transforms, while Transform is the base class for specific transformations.
• docutils.languages: Language-dependent strings and mappings. Use get_language(language_code) to retrieve the matching module.

PEP 817 (Wheel Variants) aims to solve packaging limitations in scientific computing and AI/ML by introducing hardware awareness to the wheel format. Currently, users often have to manually specify hardware-dependent variants (e.g., pip install jax[cuda13]) or manage separate package names for different architectures (e.g., xgboost-cpu).

Key improvements addressed by this PEP include:

• Automatic Hardware Selection: Enabling commands like pip install jax to automatically select the correct package matching the user's hardware.
• Simplified Installation: Reducing the need for complex dispatching logic and separate requirements.txt files for CPU vs. GPU versions.
• Reduced Binary Size: Moving away from "fat binaries" that bundle multiple GPU targets (SMs) into a single wheel.
• Improved Reproducibility: Making it easier to deploy research code across heterogeneous computing clusters (CPU, GPUs, ASICs) without environment-specific manual procedures.