S3PyPI

If you are upgrading from version 0.x (which used CloudFormation), you must import your existing resources into Terraform and rename index files to support the new CloudFront-to-S3-REST-API architecture.

1. Import resources into Terraform:

$ terraform init
$ terraform import module.s3pypi.aws_s3_bucket.pypi example-bucket
$ terraform import module.s3pypi.aws_cloudfront_distribution.cdn EDFDVBD6EXAMPLE
$ terraform apply

2. Rename index objects: Because the new configuration uses the S3 REST API instead of the S3 website endpoint, all <package>/index.html objects must be renamed to <package>/. Use the provided migration script:

$ scripts/migrate-s3-index.py example-bucket

Note: If you prefer to keep using the old publicly accessible S3 website endpoint, use the --index.html flag and set --s3-put-args='ACL=public-read' during upload.

If you enabled enable_basic_auth in your Terraform configuration, you can manage users using the basic_auth/put_user.py script. This script creates a parameter in AWS Systems Manager Parameter Store for the specified domain and user, storing a hashed password and salt.

$ basic_auth/put_user.py pypi.example.com alice
Password:

Install the s3pypi CLI tool using pip to begin managing your private Python package repository.

$ pip install s3pypi

You can install packages from your private repository using pip by specifying the --extra-index-url. If you used a --prefix during upload, include it in the URL.

Via CLI:

$ pip install your-project --extra-index-url https://pypi.example.com/PREFIX/

Via pip configuration (~/.pip/pip.conf):

[global]
extra-index-url = https://pypi.example.com/PREFIX/

To distribute your packages, first build your distribution files (e.g., using sdist and bdist_wheel), then use the s3pypi upload command to push them to your S3 bucket.

$ cd /path/to/your-project/
$ python setup.py sdist bdist_wheel

$ s3pypi upload dist/* --bucket example-bucket [--prefix PREFIX]

To host your repository, you need an S3 bucket for storage and a CloudFront distribution for HTTPS. Use the provided Terraform configuration in the terraform/ directory to provision these resources.

1. Clone the repository.
2. Navigate to the terraform/ directory.
3. Run terraform init and terraform apply.

During terraform apply, you will be prompted for your AWS region, S3 bucket name, and CloudFront domain name. Alternatively, you can provide these in a config.auto.tfvars file.

$ git clone https://github.com/gorilla-co/s3pypi.git
$ cd s3pypi/terraform/

$ terraform init
$ terraform apply

You can integrate S3PyPI into your own infrastructure code by using it as a Terraform module. This allows you to manage the S3 bucket, CloudFront distribution, and associated security settings (like WAF) alongside your existing resources.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

module "s3pypi" {
  source = "github.com/gorilla-co/s3pypi//terraform/modules/s3pypi"

  bucket = "example-bucket"
  domain = "pypi.example.com"

  use_wildcard_certificate = true
  enable_dynamodb_locking  = true
  enable_basic_auth        = true

  providers = {
    aws.us_east_1 = aws.us_east_1
  }
}

The LockerConfig dataclass controls how the DynamoDBLocker behaves when it encounters a lock already held by another process.

from s3pypi.locking import LockerConfig

# Wait 5 seconds between attempts, try up to 20 times
cfg = LockerConfig(retry_delay=5, max_attempts=20)

The s3pypi CLI requires specific IAM permissions to interact with S3 and DynamoDB (if locking is enabled). Ensure the identity running the CLI has the following policy (replace example-bucket with your actual bucket name):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::example-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::example-bucket"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:*:*:table/example-bucket-locks"
    }
  ]
}