s3fs Documentation

S3FS is built on aiobotocore and supports asynchronous operations. To use it in async code:

1. Pass asynchronous=True to the S3FileSystem constructor.
2. Explicitly await s3.set_session() before making calls.
3. Use the async versions of methods (which typically have a _ prefix, though the documentation notes sync versions exist with the same name).

Note: For non-async code, the library handles internal async operations (like bulk cp or mv) behind a synchronization layer automatically.

import asyncio
from s3fs import S3FileSystem

async def run_program():
    s3 = S3FileSystem(..., asynchronous=True)
    session = await s3.set_session()
    # ... perform work ...
    await session.close()

asyncio.run(run_program())

Use s3fs.S3FileSystem to interact with S3. For public buckets, initialize with anon=True. You can use methods like ls() to list contents and open() to get a file-like object for reading or writing.

import s3fs
s3 = s3fs.S3FileSystem(anon=True)
s3.ls('my-bucket')
with s3.open('my-bucket/my-file.txt', 'rb') as f:
    print(f.read())

S3FS can authenticate using several methods:

• Explicitly: Pass key and secret to the S3FileSystem constructor.
• Boto Auto-discovery: Rely on environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), AWS config files (~/.aws/credentials), or EC2 IAM roles.
• Profiles: Specify a profile via S3FileSystem(profile='PROFILE').
• Anonymous: Use anon=True for public, read-only access.

You can install s3fs using pip from the Python Package Index (PyPI).

pip install s3fs

You can install the s3fs library and its dependencies from the conda-forge repository using the conda package manager.

$ conda install s3fs -c conda-forge

You can enable S3 server-side encryption by passing encryption parameters via the s3_additional_kwargs argument when initializing S3FileSystem or when calling methods like s3.open.

Arguments passed to s3_additional_kwargs are appended to all underlying S3 calls. If you provide both s3_additional_kwargs at the filesystem level and specific **kwargs in a method call (like s3.open), the method-level arguments take precedence as they are applied last.

For convenience, you can use s3.utils.SSEParams instead of a raw Python dictionary to manage these parameters.

import s3fs

# Apply encryption to all calls made by this filesystem instance
s3 = s3fs.S3FileSystem(
    s3_additional_kwargs={'ServerSideEncryption': 'AES256'}
)

# You can also pass arguments directly to open()
# fo = s3.open('path/to/file', ServerSideEncryption='AES256')

To set up a local development environment for s3fs, install the necessary dependencies using pip from the project root. This includes both the core requirements and the requirements specifically needed for running tests.

$ pip install -r requirements.txt -r test_requirements.txt

Libraries like pandas, dask, and intake support S3 URLs (e.g., s3://bucket/path). You can pass configuration directly to s3fs using the storage_options argument.

import pandas as pd
# Pass s3fs arguments via storage_options
df = pd.read_excel("s3://bucket/path/file.xls", storage_options={"anon": True})

Once the development environment is set up, you can execute the test suite using pytest from the project root.

$ pytest

To use S3-compatible services, you must provide an endpoint_url. You can also pass client_kwargs or config_kwargs for provider-specific requirements (like region names or signature versions).

import s3fs

# MinIO example
s3 = s3fs.S3FileSystem(key='miniokey...', secret='asecretkey...', endpoint_url='https://...')

# Scaleway example with region
s3 = s3fs.S3FileSystem(
    key='scaleway-api-key...', 
    secret='scaleway-secretkey...', 
    endpoint_url='https://s3.fr-par.scw.cloud', 
    client_kwargs={'region_name': 'fr-par'}
)

# OVH example with specific signature version
s3 = s3fs.S3FileSystem(
    key='ovh-s3-key...', 
    secret='ovh-s3-secretkey...', 
    endpoint_url='https://s3.GRA.cloud.ovh.net', 
    client_kwargs={'region_name': 'GRA'},
    config_kwargs={'signature_version': 's3v4'}
)