DOCSEARCHbyAlgolia
Submit Docs

s3-credentials

repository·main·Indexed 42 days ago

https://github.com/simonw/s3-credentials

A tool for creating scoped AWS credentials for accessing specific S3 buckets. Version 0.17 allows for the generation of permanent or temporary credentials (via STS.AssumeRole), bucket creation, and fine-grained permission control (read-only or write-only). It includes utilities for managing S3 objects, inspecting bucket policies, and a localserver to serve temporary credentials via GET requests.

Tokens
13.2K
Snippets
74
Records
78
Agent score
51%

What's inside s3-credentials

  1. How credential caching works in the local server

    main
    The local server caches credentials internally and serves the same credentials until they expire. When the duration elapses, the server automatically generates new credentials. This mechanism prevents the "dogpile effect" (multiple simultaneous requests triggering multiple credential generations) and ensures that all applications fetching credentials within a short window receive identical credentials.

  2. Create permanent S3 credentials

    main

    To create permanent credentials that never expire, run s3-credentials create followed by one or more bucket names. The tool will create a dedicated AWS IAM user, attach a policy for the specified buckets, and return an access key.

    Important: Record the SecretAccessKey immediately, as it is only displayed once and cannot be recovered.

    s3-credentials create static.niche-museums.com

  3. Create new AWS credentials for S3 buckets

    main

    The create command generates new AWS credentials for specified S3 buckets. It can optionally create the bucket if it doesn't exist and allows for fine-grained control over permissions (read-only, write-only) and credential duration.

    Common Tasks:

    • Create a new bucket with read-write credentials: s3-credentials create my-new-bucket -c
    • Create read-only credentials for an existing bucket: s3-credentials create my-existing-bucket --read-only
    • Create temporary write-only credentials (e.g., valid for 15 minutes): s3-credentials create my-existing-bucket --write-only -d 15m

    Key Options:

    • -f, --format [ini|json]: Output format for credentials.
    • -d, --duration DURATION: How long credentials work (e.g., 3600, 15m, 1h). Defaults to forever.
    • --username TEXT: Specific username to create or use.
    • -c, --create-bucket: Create the bucket if it does not exist.
    • --read-only: Restrict credentials to reading only.
    • --write-only: Restrict credentials to writing only.
    • --policy POLICY: Path to a policy.json file or a literal JSON string. Use $!BUCKET_NAME!$ as a placeholder for the bucket name.
    • --endpoint-url TEXT: Custom endpoint URL (useful for S3-compatible storage).
    • -a, --auth FILENAME: Path to a JSON/INI file containing existing credentials.
    s3-credentials create my-new-bucket -c

  4. Download objects from an S3 bucket

    main

    Use get-object for single files or get-objects for multiple files or entire buckets.

    Single Object (get-object):

    • Output to stdout: s3-credentials get-object my-bucket hello.txt
    • Save to a specific file: s3-credentials get-object my-bucket hello.txt -o hello.txt

    Multiple Objects (get-objects):

    • Download everything in a bucket: s3-credentials get-objects my-bucket (saves to a directory named after the bucket).
    • Download specific keys: s3-credentials get-objects my-bucket one.txt path/two.txt
    • Download using glob patterns: s3-credentials get-objects my-bucket --pattern '*/*.js'

    Key Options for get-objects:

    • -o, --output DIRECTORY: Specify a custom directory for downloads.
    • -p, --pattern TEXT: Glob pattern for filtering files.
    • -s, --silent: Hide the progress bar.
    s3-credentials get-objects my-bucket --pattern '*/*.js'

  5. Start the local credential server

    main

    The s3-credentials localserver command starts a local HTTP server that serves temporary S3 credentials in the AWS credential_process JSON format. This is useful for providing credentials to applications that can fetch them from an HTTP endpoint.

    By default, the server starts on localhost:8094 and provides read-write access to the specified bucket.

    s3-credentials localserver my-bucket --duration 1h

  6. Create temporary S3 credentials

    main

    To create temporary credentials using STS.AssumeRole(), use the --duration option. The duration must be between 15 minutes and 12 hours.

    Note: When using temporary credentials, you must use the AccessKeyId, SecretAccessKey, AND the SessionToken provided in the output.

    s3-credentials create static.niche-museums.com --duration 15m

  7. Configure AWS CLI to use the local credential server

    main

    You can integrate the local server with the AWS CLI by configuring a profile in your ~/.aws/config file using the credential_process setting. This allows the AWS CLI to automatically fetch and refresh credentials from your local server.

    [profile localserver]
credential_process = curl -s http://localhost:8094/

    Then use the profile with any AWS command:

    aws s3 ls s3://my-bucket/ --profile localserver

  9. Configure AWS credentials for s3-credentials

    main

    The s3-credentials tool uses boto3 for AWS authentication. You can provide credentials using any of the following methods:

    1. AWS Configuration Files: The tool automatically detects existing ~/.aws/config or ~/.aws/credentials files. You can create these using the aws configure command from the awscli package.
    2. Environment Variables: Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your shell environment before running the tool.
    3. Command Line Options: Pass credentials directly using flags like --access-key and --secret-key.
    4. Auth Files: Provide a file containing credentials via the --auth option.

  10. Generate IAM policy documents for an S3 bucket

    main

    Use the policy command to generate IAM policy JSON documents for a specific S3 bucket. This allows you to preview the permissions that will be granted to the credentials generated by the tool. By default, the tool generates a read-write policy. You can modify the scope and permissions using flags like --read-only, --write-only, --prefix, and --public-bucket.

    s3-credentials policy <bucket-name>

  11. Delete objects from an S3 bucket

    main

    The delete-objects command removes one or more objects from a specified bucket.

    Common Tasks:

    • Delete specific keys: s3-credentials delete-objects my-bucket one.txt two.txt
    • Delete all files matching a prefix: s3-credentials delete-objects my-bucket --prefix my-folder/

    Key Options:

    • --prefix TEXT: Delete everything starting with this prefix.
    • -d, --dry-run: Show which keys would be deleted without actually performing the deletion.
    • -s, --silent: Suppress informational output.
    s3-credentials delete-objects my-bucket --prefix my-folder/

  12. Use a custom policy with s3-credentials

    main

    You can provide a custom IAM policy using the --policy option. If your policy uses the placeholder $!BUCKET_NAME!$, the tool will automatically replace it with the actual S3 bucket name before applying it.

    You can provide the policy via:

    1. A file path.
    2. A literal JSON string.
    3. Standard input (-).
    s3-credentials create my-s3-bucket --policy custom-policy.json