cloudpathlib

The AnyPath constructor follows a specific fallback logic to resolve the input type:

1. Cloud Check: It first attempts to pass the input to the CloudPath base class constructor. This validates the input against registered concrete CloudPath implementations (e.g., S3Path for s3:// prefixes).
2. Local Fallback: If no cloud implementation matches, it attempts to pass the input to the standard pathlib.Path constructor.
3. Error Handling: If the Path constructor fails and raises a TypeError, AnyPath will raise an AnyPathTypeError exception.

The cloudpathlib.AnyPath class provides a unified interface for interacting with files across different storage backends (such as local filesystems, AWS S3, Google Cloud Storage, etc.). It acts as a factory or a polymorphic path object that dispatches operations to the appropriate backend implementation based on the path prefix (e.g., s3:// for AWS S3 or a local path for the local filesystem). This allows you to write code that is agnostic of the underlying storage provider.

from cloudpathlib import AnyPath

# AnyPath automatically detects the backend based on the prefix
path = AnyPath("s3://my-bucket/data/file.txt")
# or
path = AnyPath("data/file.txt")

Communication with cloud services is handled by Client objects (S3Client, AzureBlobClient, or GSClient). A client holds the authenticated connection and local cache configuration.

When you instantiate a cloud path (e.g., S3Path) for the first time, a default client is created automatically. All subsequent paths for that service will share the same client instance. To use custom authentication or configurations, you must explicitly instantiate a client and use it to create paths.

from cloudpathlib import CloudPath

# A default client is created automatically
cloud_path = CloudPath("s3://cloudpathlib-test-bucket/")
print(cloud_path.client)

The AnyPath class allows you to write code that seamlessly handles both local filepaths and cloud storage paths without using conditional logic. When you instantiate AnyPath with a string, it automatically determines whether to return a pathlib.Path instance (for local paths) or a specific CloudPath instance (like S3Path) based on the URI scheme provided.

Because AnyPath acts as a virtual superclass for both CloudPath and Path, isinstance(obj, AnyPath) will return True for both local and cloud path objects.

from cloudpathlib import AnyPath

# Automatically becomes a local PosixPath
path = AnyPath("mydir/myfile.txt")

# Automatically becomes an S3Path
cloud_path = AnyPath("s3://mybucket/myfile.txt")

# Polymorphic checks work
isinstance(path, AnyPath)      # True
isinstance(cloud_path, AnyPath) # True

For standard use, cloudpathlib automatically reads credentials from standard environment variables used by each cloud service's SDK. This is the recommended approach for security and ease of use.

AnyPath supports the file: URI scheme. When provided with a file: URI, it returns a standard pathlib.Path instance. You can use the standard .as_uri() method on the resulting path object to convert it back into a file: URI after performing path manipulations.

from cloudpathlib import AnyPath

# Supports hostname omitted variant
path = AnyPath("file:/root/mydir/myfile.txt")

# Supports explicit local path variant
path = AnyPath("file:///root/mydir/myfile.txt")

# Convert back to file:// URI after manipulation
parent_uri = path.parent.as_uri()
# Result: 'file:///root/mydir'

You can install cloudpathlib via conda-forge. To include specific cloud SDKs, append the service suffix to the package name (e.g., cloudpathlib-s3). If no suffix is used, only the base classes will be available.

# Install with S3 support
conda install cloudpathlib-s3 -c conda-forge

You can use cloudpathlib path classes (like S3Path) as type annotations in Pydantic models. When a model is instantiated with a string URI, Pydantic will automatically run that input through the cloud path's constructor, converting the string into a proper path object.

This also works with the AnyPath polymorphic class. When using AnyPath, the input string is dispatched and instantiated as the appropriate specific class (e.g., S3Path for S3 URIs or PosixPath for local paths).

from cloudpathlib import AnyPath
from pydantic import BaseModel

class FancyModel(BaseModel):
    path: AnyPath

fancy1 = FancyModel(path="s3://mybucket/myfile.txt")
# fancy1.path is an S3Path

fancy2 = FancyModel(path="mydir/myfile.txt")
# fancy2.path is a PosixPath

By default, Pydantic serializes cloudpathlib objects using their URI representation. If you need a different format (for example, a full HTTPS URL instead of an s3:// URI), you can use Pydantic's Annotated and PlainSerializer to define a custom serialization logic.

from typing import Annotated
from cloudpathlib import S3Path
from pydantic import BaseModel, PlainSerializer

class MyModel(BaseModel):
    # Uses as_url() to serialize to a web URL instead of the default URI
    s3_file: Annotated[S3Path, PlainSerializer(lambda x: x.as_url())]

inst = MyModel(s3_file="s3://mybucket/myfile.txt")
print(inst.model_dump_json())
# Output: '{"s3_file":"https://mybucket.s3.amazonaws.com/myfile.txt"}'

To access public S3 buckets without providing credentials, instantiate an S3Client with no_sign_request=True. Without this, cloudpathlib will attempt to use environment credentials and throw a NoCredentialsError.

Note: Many public buckets do not allow directory listing for anonymous users. In such cases, you can only interact with CloudPath objects that point directly to specific files.

from cloudpathlib import S3Client

# Create a client for anonymous access
c = S3Client(no_sign_request=True)

# Use the client to access the public file
path = c.CloudPath("s3://ladi/Images/FEMA_CAP/2020/70349/DSC_0001_5a63d42e-27c6-448a-84f1-bfc632125b8e.jpg")
print(path.exists())

To connect to a custom S3-compatible service, instantiate S3Client with the endpoint_url parameter (including the protocol and port). If the service requires virtual-hosted-style URLs, set addressing_style="virtual".

from cloudpathlib import S3Client, CloudPath

# Create a client pointing to a custom endpoint
client = S3Client(endpoint_url="http://my.s3.server:1234", addressing_style="virtual")

# Use the client to create paths
cp = client.CloudPath("s3://my-custom-bucket/")

To handle authentication (like Basic Auth), pass a urllib.request.BaseHandler implementation to the HttpClient or HttpsClient constructor. This allows you to manage credentials for specific realms and URIs.

import urllib.request
from cloudpathlib import HttpClient

auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(
    realm="Some Realm",
    uri="http://www.example.com",
    user="username",
    passwd="password"
)

client = HttpClient(auth=auth_handler)
my_path = client.CloudPath("http://www.example.com/secret/data.txt")

# Now GET requests will include basic auth headers
content = my_path.read_text()