DOCSEARCHbyAlgolia
Submit Docs

PyPI Warehouse

repository·main·Indexed 26 days ago

https://github.com/pypi/warehouse

The codebase for PyPI.org, including documentation for the /api/* namespace, Docker Compose development environment configuration, and guides for managing the PyPI Blog and user-facing documentation. It covers system components such as the web application, background workers, and image proxy (Camo), as well as administrative details regarding user identity, project journals, and security enforcement like 2FA and Trusted Publishing.

Tokens
56K
Snippets
97
Records
387
Agent score
88%

What's inside pypi-warehouse

  1. Overview of PyPI UI changes and rollout phases

    main

    The PyPI UI is undergoing a multi-phase redesign to improve security transparency (specifically for digital attestations) and information hierarchy.

    Rollout Phases:

    1. Phase 1: Project Details (Live on TestPyPI) - Redesign of the main page, including sidebar reordering and repositioning.
    2. Phase 2: Files and Release History - Streamlining tabs and adding attestation metadata.
    3. Phase 3: Security Tab - Introduction of a dedicated space for provenance and attestation data with security warnings.
    4. Phase 4: Documentation Refresh - Updating PyPI documentation to align with the new UI and security guidance.

    Key UI Changes:

    • Dedicated Security Tab: Centralizes provenance and attestation metadata.
    • Horizontal Navigation: Moves internal navigation from the sidebar to horizontal tabs.
    • Right-aligned Metadata Sidebar: Moves the sidebar to the right to prioritize the package description/readme on the left.
    • Trust Level Labeling: Explicitly distinguishes between PyPI-intrinsic data (e.g., timestamps) and maintainer-provided/verified data (e.g., upstream source repositories).
    • Improved Status Labeling: Uses bolder colors for quarantined, yanked, archived, and pre-release states.

  3. Overview of Cabotage

    main

    Cabotage is a 'Platform as a Service' (PaaS) designed to deploy arbitrary services into Kubernetes clusters in a 'Twelve-Factor' style without requiring manual YAML configuration. It is used to deploy PyPI and its supporting services (such as conveyor, camo, and inspector).

    Key features include:

    • End-to-end TLS.
    • Protection against recovering secrets through the web UI.
    • Isolation between tenants within the Kubernetes cluster.
    • Automated deployments and self-service configuration for project administrators.

  4. Understand the Warehouse System Architecture

    main

    Warehouse is a distributed system composed of web services and background workers.

    Key Interaction Patterns:

    • Standard Traffic: End users interact with Warehouse via Fastly (CDN), which proxies traffic to the origin. Package files are served via URLs prefixed with https://files.pythonhosted.org/packages/... and are cached by Fastly.
    • Uploads: When uploading files, Fastly is bypassed; uploads go directly to the origin Warehouse instance via HTTPS.
    • Storage: Warehouse uses Backblaze B2 as primary storage (to leverage waived egress fees with Fastly) and AWS S3 as an archive/fallback storage.
    • Search: Project and package metadata are indexed in OpenSearch.

  6. Understand the PyPI Attestation Security Model

    main

    PyPI attestations assert facts about a package, primarily that it was published by an authorized publisher (such as a specific CI provider in a specific code repository).

    Key purposes of attestations:

    • Post-build protection: Protects against modification of a project after it was built (e.g., while stored in a mirror).
    • Change observation: Allows verifying parties to detect changes to a project's Trusted Publisher, which can indicate a malicious takeover of the project identity.

  7. How Trusted Publishing works via OIDC

    main

    PyPI's Trusted Publishing uses OpenID Connect (OIDC) to allow services (like GitHub Actions) to identify themselves without manual token management.

    The workflow consists of three main components:

    1. OIDC Identity Providers (Providers): Services like GitHub generate OIDC tokens containing scoped claims. For example, a repo claim might be set to octo-org/example.
    2. Trusted Publishers Configuration: Configuration on PyPI that defines which OIDC providers to trust and which specific claims must be present (e.g., matching a specific repo, workflow, and environment). PyPI also verifies claims like repository_owner_id to prevent account resurrection attacks.
    3. Token Exchange: PyPI verifies the OIDC token's signature and matches its claims against registered Trusted Publishers. If a match is found, PyPI issues a short-lived (15 minute) PyPI API token. This token is scoped to all projects associated with that matching Trusted Publisher configuration.

  8. Understand Integrity API concepts: Attestations vs Provenance

    main

    The Integrity API implements [PEP 740] using two primary object types:

    • Attestation objects: Encapsulate a single attestation for a single file (e.g., a publish attestation or SLSA Provenance).
    • Provenance objects: Encapsulate one or more attestations for a given file, bundling them together with the identity that produced them.

    Usage Note: The API returns provenance objects. Users are responsible for extracting and verifying individual attestations from the provenance object as needed.

  10. Understand the PyPI-specific JSON API

    main

    The PyPI-specific JSON API provides access to detailed project and release information.

    Note: If your goal is simply to retrieve all distributions or all versions for a package (e.g., for automated downloading), you should use the [Index API] instead of this JSON API.

  11. Understand PyPI storage limits

    main

    PyPI imposes two types of storage limits:

    1. File size limit: The maximum size for an individual file upload. The default is 100.0 MB.
    2. Project size limit: The maximum total size of all files within a single project. The default is 10.0 GB.

    You can monitor your project's current size and limits on your project settings page at https://pypi.org/manage/project/YOUR-PROJECT/settings/.

  12. Understand the Warehouse technology stack

    main

    Warehouse is built using the following technologies:

    Key Architectural Notes for Developers:

    • Hybrid URL Traversal and Dispatch: Warehouse uses Pyramid's hybrid approach where factory classes provide resources directly to views based on URL patterns. This differs from the standard URL dispatch used in Flask or Django.
    • ORM Implementation: Because Warehouse was built on top of a legacy database, some SQLAlchemy logic may use name-based joins instead of standard foreign keys.