bandersnatch

bandersnatch is a PyPI mirror client that implements PEP 381. It uses the PEP 691 Simple API to fetch package serial metadata and the PyPI JSON API to retrieve shasums and release file paths. It then manages the layout of these package files on a POSIX file system.

Key technical characteristics:

• Asyncio-based: The core logic is built on asyncio (primarily using aiohttp).
• PEP 691 Support: Supports both HTML and JSON Simple Index formats.
• Normalization: It stores PEP 503 normalized package names for the /simple API and stores JSON in normalized package name paths.

The bandersnatch_filter_plugins package provides a collection of submodules designed to filter packages during the bandersnatch synchronization process. These plugins allow you to include or exclude packages based on various criteria such as names, filenames, metadata, or regex patterns.

Available filtering modules include:

• blocklist_name: Filter packages using a blocklist of names.
• allowlist_name: Filter packages using an allowlist of names.
• filename_name: Filter based on filename patterns.
• latest_name: Filter to keep only the latest versions.
• metadata_filter: Filter based on package metadata.
• prerelease_name: Filter out or include prereleases.
• regex_name: Filter using regular expressions.

When the json option is enabled, Bandersnatch saves PyPI project metadata in two locations. For a project with the normalized name someproject, the JSON metadata is stored in:

1. <mirror directory>/web/json/someproject
2. <mirror directory>/pypi/someproject/json

<mirror directory>/\n├── web/\n│   └── json/\n│       └── someproject\n├── pypi/\n│   ├── someproject/\n│   │   └── json\n├── packages/\n└── simple/

When mirroring from an existing index, the upstream mirror must support the following:

1. The Simple Repository API with PEP 691 JSON responses: It must respond to the Accept: application/vnd.pypi.simple.v1+json header with a Content-Type of application/vnd.pypi.simple.v1+json.
2. The PyPI JSON metadata endpoint: The endpoint at /pypi/<project>/json must be available.

Bandersnatch relies on these JSON formats for correct mirroring and will fail if only HTML (PEP 503) is provided.

The simple-format setting determines how index files are generated in the <mirror directory>/web/simple/ directory. The structure is compatible with the Simple Repository API.

• If simple-format is set to HTML: Bandersnatch creates index.html and index.v1_html files.
• If simple-format is set to JSON: Bandersnatch creates index.v1_json files.

<mirror directory>/\n└── web/\n    ├── packages/...\n    └── simple/\n        ├── index.html\n        ├── index.v1_html\n        ├── index.v1_json\n        ├── someproject/\n        │   ├── index.html\n        │   ├── index.v1_html\n        │   └── index.v1_json\n        └── ...

Package release files are distributed into subdirectories based on their checksums within the <mirror directory>/web/packages/ directory. By default, the generated index files contain relative links pointing into this web/packages/ directory.

<mirror directory>/\n└── web/\n    ├── packages/\n    │   ├── 1a/\n    │   │   └── 70/\n    │   │       └── e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/\n    │   │           └── click-8.1.6-py3-none-any.whl\n    │   └── ...\n    └── simple/\n        ├── click/\n        └── index.html

You can select packages or release files by filtering on specific metadata values using a custom configuration section. The general format for an entry is:

tag:tag:path.to.object = match_value

Valid tags for the prefix are:

• all
• any (default)
• none
• match-null
• not-null

All metadata provided by the PyPI JSON API is available, including info, last_serial, and releases.

# Example: Filter projects by Python version classifier
[regex_project_metadata]
not-null:info.classifiers = 
    .*Programming Language :: Python :: 2.*

# Example: Filter release files by package type
[regex_release_file_metadata]
any:release_file.packagetype =
    sdist
    bdist_wheel

The hash-index option groups project index folders by the first letter of their normalized name (e.g., /simple/s/someproject/). This improves filesystem efficiency for very large mirrors but makes the structure incompatible with the standard Simple Repository API.

Warning: You cannot serve the web/simple/ folder directly with pip. You must use a web server (like Apache or NGINX) that can rewrite URLs to match the new filesystem structure.

NGINX Rewrite Example:

rewrite ^/simple/([^/])([^/]*)/$ /simple/$1/$1$2/ last;
rewrite ^/simple/([^/])([^/]*)/([^/]+)$/ /simple/$1/$1$2/$3 last;

When hash-index is enabled, project index folders are grouped by the first letter of their name within the <mirror directory>/web/simple/ directory. The content of the index files themselves remains unchanged.

<mirror directory>/\n└── web/\n    └── simple/\n        ├── b/\n        │   ├── boto3/\n        │   │   └── index.html\n        │   └── botocore/\n        │       └── index.html\n        ├── c/\n        │   ├── certifi/\n        │   │   └── index.html\n        │   └── cryptography/\n        │       └── index.html\n        └── index.html

Follow these steps to initialize and populate a mirror:

1. Initialize configuration: Run bandersnatch mirror. This will create an empty configuration file at /etc/bandersnatch.conf.
2. Configure: Review and adapt /etc/bandersnatch.conf to your requirements (e.g., adding a blocklist or allowlist to control mirror size).
3. Populate: Run bandersnatch mirror again. This will download and populate your mirror with the current status of PyPI packages.
4. Maintain: Run bandersnatch mirror regularly to keep the mirror updated.

If you have an existing mirror that was created before core metadata support, you can backfill missing .metadata files using one of two methods:

1. Full Sync (--force-check): Run bandersnatch mirror --force-check. This performs a complete sync: it skips current release files but downloads missing core metadata and regenerates all simple index files to advertise them.
2. Metadata Refresh (--json-update): Run bandersnatch verify --json-update. This refreshes saved JSON metadata and downloads missing/invalid metadata files. Note: This requires json = true in your config. This method repairs package files but does not regenerate simple index files; index files will only reflect the new metadata once the packages are re-synced via bandersnatch mirror.

To run Bandersnatch using Docker Compose, follow these steps:

1. Preparation:
   • Configure your Bandersnatch settings in conf/bandersnatch.conf.
   • Adjust the volume mappings in the docker-compose.yml file to match your local paths.
   • MTU Configuration: Adjust the MTU in the compose file. While the example uses 800 for compatibility, 1500 is a common standard. If you are using nested virtualization or software-defined networks, you may need to lower it; if your network supports jumbo frames, you may want to increase it.
2. Pull the Image: Pull the latest official image.
3. Run: Start the containers in detached mode.

# Pull the image
docker pull pypa/bandersnatch:latest

# Run the containers
docker compose up -d