DOCSEARCHbyAlgolia
Submit Docs

s5cmd Documentation

repository·master·Indexed 26 days ago

https://github.com/peak/s5cmd

s5cmd is a high-performance S3 and local filesystem execution tool designed for extreme speed in object storage workflows. It supports wildcard operations, command file batching via the 'run' command, S3 Transfer Acceleration, and compatibility with Google Cloud Storage and other S3-compatible services. Key features include a 'sync' command for directory synchronization, SQL queries against JSON objects via 'select', and configurable parallelism using '--numworkers' and '--concurrency' flags.

Tokens
9.7K
Snippets
15
Records
81
Agent score
88%

What's inside s5cmd

  1. Overview of s5cmd

    master
    s5cmd is a high-performance execution tool for S3 and local filesystems. It is designed for speed, significantly outperforming s3cmd and aws-cli in both upload and download operations. It supports a wide range of object management tasks including wildcard support, command file batching, and S3 Transfer Acceleration. It is also compatible with Google Cloud Storage and other S3-compatible services.

  2. Configure s5cmd credentials

    master

    s5cmd uses the official AWS SDK for authentication. You can provide credentials using several methods:

    Command Line Flags

    • --profile <name>: Use a specific named profile.
    • --credentials-file <path>: Use a specific credentials file.

    Environment Variables

    Set the following variables to configure access:

    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • AWS_PROFILE
    • AWS_REGION
    • S3_ENDPOINT_URL (for non-AWS S3 compatible services)

    IAM Roles

    If running on AWS infrastructure, s5cmd automatically supports:

    • EC2 IAM roles
    • EKS Kube IAM roles

    Anonymous Access

    Use the --no-sign-request flag to access public buckets without credentials.

    # Use a specific profile from the default AWS credentials file
s5cmd --profile my-work-profile ls s3://my-company-bucket/

# Use a custom credentials file and profile
s5cmd --credentials-file ~/.your-credentials-file --profile my-work-profile ls s3://my-company-bucket/

# Use environment variables
export AWS_ACCESS_KEY_ID='<your-access-key-id>'
export AWS_SECRET_ACCESS_KEY='<your-secret-access-key>'
export AWS_PROFILE='<your-profile-name>'
export AWS_REGION='<your-bucket-region>'

# Anonymous access for public buckets
s5cmd --no-sign-request ls s3://public-bucket/

  3. Run s5cmd performance regression tests with bench.py

    master

    The bench.py script compares the performance of two different s5cmd builds (e.g., version tags, PR numbers, or commit tags) across various scenarios like uploading, downloading, and removing small, large, or very large files.

    Note on Scenarios: To change the file sizes or counts used in the scenarios, you must currently edit the bench.py script directly.

  5. Common s5cmd operations

    master
    Perform standard S3 operations including head, copy, upload, delete, and more. s5cmd supports multiple-level wildcards (e.g., *) for all operations, which are processed by listing objects with the prefix and filtering in-memory.

  6. Run s5cmd using Docker

    master
    You can pull the official image from Docker Hub and run operations. To use your local AWS credentials, mount your ~/.aws directory. To run operations against your current local directory, mount it to the /aws working directory inside the container.

  8. Use extra flags with bench.py

    master

    When passing extra flags for hyperfine (-hf) or s5cmd (-sf), you must wrap the flags in quotation marks and start the string with a space. Failure to include the leading space may cause errors due to how argparse processes subprocess arguments.

    ./bench.py --bucket tempbucket --s5cmd v2.0.0 456 --warmup 2 --runs 10 -sf " --log error" -hf " --show-output"

  9. Run multiple commands in parallel with `s5cmd run`

    master

    Execute thousands of S3 or filesystem commands efficiently by providing a command file. This avoids the overhead of multiple fork-exec calls by using parallel workers within a single process.

    # Run commands from a file
s5cmd run commands.txt

# Pipe commands into s5cmd run
cat commands.txt | s5cmd run

    Example commands.txt content:

    cp 's3://bucket/2020/03/*' logs/2020/03/
# line comments are supported
rm s3://bucket/2020/03/19/file2.gz
mv s3://bucket/2020/03/18/file1.gz s3://bucket/2020/03/18/original/file.gz

  10. Install shell auto-completion

    master

    s5cmd supports auto-completion for bash, zsh, and pwsh.

    1. Run s5cmd --install-completion to get the instructions.
    2. For bash/zsh: Add the provided script to your .bashrc or .zshrc.
    3. For pwsh: Save the script as s5cmd.ps1 and add its full path to your $profile.
    4. Ensure the SHELL environment variable is correctly set to your shell binary.

  12. Install s5cmd via Conda

    master

    You can install s5cmd from the conda-forge channel. First, ensure the channel is added and priority is set, then install the package.

    conda config --add channels conda-forge
conda config --set channel_priority strict
conda install s5cmd