DOCSEARCHbyAlgolia
Submit Docs

rust-s3

repository·master·Indexed 19 days ago

https://github.com/durch/rust-s3

A flexible Rust library for interacting with Amazon S3 and S3-compatible object storage APIs such as Minio, R2, Google Cloud Storage, and Backblaze B2. It supports async/await (via tokio or async-std) and synchronous execution models. The library includes the aws-creds crate for managing IAM credentials from environment variables, credential files, and STS, as well as the aws-region crate for defining AWS S3 region identifiers and custom endpoints.

Tokens
21K
Snippets
76
Records
98
Agent score
72%

What's inside rust-s3

  2. Use the aws-region crate to define S3 regions

    master

    The aws-region crate provides a Region type to represent AWS S3 region identifiers. You can create a Region instance by parsing a string, using a predefined enum variant for standard AWS regions, or by defining a Custom region with a specific name and endpoint.

    Important: When using Region::Custom, you must provide a valid endpoint. Providing an invalid endpoint for a custom region will result in errors during operation.

    use std::str::FromStr;
use awsregion::Region;

// Parse from a string
let region: Region = "us-east-1".parse().unwrap();

// Choose region directly
let region = Region::EuWest2;

// Custom region requires valid region name and endpoint
let region_name = "nl-ams".to_string();
let endpoint = "https://s3.nl-ams.scw.cloud".to_string();
let region = Region::Custom { region: region_name, endpoint };

  3. Manage AWS credentials with aws-creds

    master

    The aws-creds crate provides utilities to load AWS access credentials (access key, secret key, and optional session token) from multiple sources.

    Credentials can be loaded in the following order of preference:

    1. Arguments: Explicitly passed to Credentials::new().
    2. Environment Variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN.
    3. AWS Credentials File: The standard AWS credentials file using a specific profile name.

    To load from the standard AWS credentials file, use Credentials::default() (which uses the [default] profile) or Credentials::new() with a profile name provided as the fourth argument.

    use awscreds::Credentials;

// Load from [default] profile in AWS credentials file
let credentials = Credentials::default();

// Load from [my-profile] profile in AWS credentials file
let credentials = Credentials::new(None, None, None, Some("my-profile".into()));

  4. How multipart uploads work in rust-s3

    master

    When uploading large files via put_object_stream, the library automatically manages multipart uploads to optimize performance and memory usage.

    1. Automatic Detection: The library reads the first chunk. If the file size is smaller than CHUNK_SIZE (8MB), it performs a standard PutObject.
    2. Multipart Initiation: If the file is large, it calls initiate_multipart_upload to get an upload_id.
    3. Bounded Parallelism: The library calculates max_concurrent_chunks based on available system memory (clamped between 2 and 100) to prevent OOM errors. It uses FuturesUnordered to upload multiple chunks concurrently.
    4. Error Handling: If any chunk upload fails, the library attempts to call abort_upload to clean up the partial upload on S3.
    5. Completion: Once all chunks are uploaded, it collects the ETags and calls complete_multipart_upload to finalize the file.

  5. Configure rust-s3 features and runtimes

    master

    The library supports multiple runtimes and execution models via Cargo features. You can choose between tokio (default), async-std, or a purely synchronous (sync) implementation.

    Key features include:

    • default: Uses tokio runtime and native-tls.
    • blocking: Generates *_blocking variants of all Bucket methods for synchronous use.
    • fail-on-err: Returns Result::Err for HTTP errors instead of treating them as successful responses.
    • with-async-std: Uses async-std runtime and surf client.
    • sync: Uses no async runtime and attohttpc for HTTP requests.
    • tags: Required to use Bucket::get_object_tagging.

    All runtimes support either native-tls or rustls-tls.

    # tokio, default
cargo run --example tokio

# async-std
cargo run --example async-std --no-default-features --features async-std-native-tls

# sync
cargo run --example sync --no-default-features --features sync-native-tls

# minio
cargo run --example minio

# r2
cargo run --example r2

# google cloud
cargo run --example google-cloud

  6. Configure AWS S3 Regions

    master

    The Region enum is used to specify the S3 region or provider endpoint. It supports standard AWS regions, DigitalOcean, Wasabi, Cloudflare R2, OVH, and Yandex. You can use predefined variants, parse from a string, or use a Custom variant for any S3-compatible provider.

    Note on Custom Regions: When using Region::Custom, you must provide both a valid region name and a valid endpoint. If you parse an unknown string using FromStr, it will automatically default to a Custom region where the region name and endpoint are identical.

    use std::str::FromStr;
use awsregion::Region;

// Parse from a string
let region: Region = "us-east-1".parse().unwrap();

// Choose region directly
let region = Region::EuWest2;

// Custom region requires valid region name and endpoint
let region_name = "nl-ams".to_string();
let endpoint = "https://s3.nl-ams.scw.cloud".to_string();
let region = Region::Custom { region: region_name, endpoint };

  7. Configure S3 bucket access with CannedBucketAcl

    master

    Use the CannedBucketAcl enum to apply standard, predefined Amazon S3 access control lists (ACLs) to a bucket. This is a convenient way to set common permission levels without defining granular user/group permissions.

    Available variants:

    • Private: Only the owner has full control.
    • PublicRead: Anyone can read objects in the bucket.
    • PublicReadWrite: Anyone can read and write objects in the bucket.
    • AuthenticatedRead: Only authenticated AWS users can read objects.
    • Custom(String): Allows specifying a custom ACL string.
    use rust_s3::CannedBucketAcl;

// Example: Setting a bucket to public read
let acl = CannedBucketAcl::PublicRead;

  8. Configure granular bucket permissions with BucketAcl

    master

    For more precise control, use the BucketAcl enum to grant specific permissions to individual users, groups, or email addresses. These can be used within a BucketConfiguration to populate grant fields like grant_read or grant_full_control.

    use rust_s3::BucketAcl;

let user_acl = BucketAcl::Id { id: "user-id-123".to_string() };
let group_acl = BucketAcl::Uri { uri: "http://example.com/group".to_string() };
let email_acl = BucketAcl::Email { email: "user@example.com".to_string() };

  9. Understand S3 object and bucket listing results

    master

    The library provides structured types for parsing S3 listing responses:

    • ListBucketResult: Represents the result of ListObjects or ListObjectsV2. It contains metadata about the bucket and a list of Objects. It handles pagination via is_truncated and next_continuation_token (or next_marker for older API versions).
    • Object: Represents an individual item in a bucket, containing the key, size, last_modified date, e_tag, and storage_class.
    • ListMultipartUploadsResult: Represents a list of ongoing multipart uploads in a bucket.

  10. Generate presigned POST URLs with PostPolicy

    master

    Use PostPolicy to create secure, conditional upload URLs for S3. You define an expiration time and a set of conditions (like file size limits or specific keys) that the upload must satisfy. Once signed, it produces a PresignedPost object containing the upload URL and the necessary form fields.

    Workflow

    1. Initialize: Create a new policy with PostPolicy::new(expiration).
    2. Add Conditions: Use .condition(field, value) to restrict the upload (e.g., limiting the file path or size).
    3. Sign: Call .sign(bucket) to generate the PresignedPost.
    4. Upload: Use the url and the combined fields and dynamic_fields from the PresignedPost to perform a multipart/form-data POST request to S3.
    // Example: Create a policy that allows uploading a file to a specific prefix
// with a maximum size of 3MB, expiring in 1 hour.
let policy = PostPolicy::new(3600)
    .condition(
        PostPolicyField::Key, 
        PostPolicyValue::StartsWith(Cow::from("uploads/user1/"))
    )?
    .condition(
        PostPolicyField::ContentLengthRange, 
        PostPolicyValue::Range(0, 3_000_000)
    )?;

let presigned_post = policy.sign(bucket).await?;

// Use presigned_post.url and the fields for your HTTP client

  11. Handle LocalStack or S3-compatible location constraints

    master

    Some S3-compatible services (like LocalStack) do not support AWS-style location constraints in bucket creation requests and may return InvalidLocationConstraint errors. You can skip sending these constraints by setting the RUST_S3_SKIP_LOCATION_CONSTRAINT environment variable to true or 1.

    export RUST_S3_SKIP_LOCATION_CONSTRAINT=true
# or
export RUST_S3_SKIP_LOCATION_CONSTRAINT=1

  12. Initialize AWS credentials directly

    master

    You can bypass environment variables and credential files by passing the keys directly to the Credentials::new constructor.

    use s3::credentials::Credentials;

let access_key = String::from("AKIAIOSFODNN7EXAMPLE");
let secret_key = String::from("wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY");

// Arguments: access_key, secret_key, session_token, profile_name
let credentials = Credentials::new(Some(access_key), Some(secret_key), None, None);