adlfs

The az:// and abfs:// protocols allow you to interact with Azure Blob Storage and ADLS Gen2. You can use these protocols with libraries like dask by passing storage_options containing your credentials.

Accepted URI formats:

• PROTOCOL://container/path-part/file
• PROTOCOL://container@account.blob.core.windows.net/path-part/file
• PROTOCOL://container@account.dfs.core.windows.net/path-part/file

import dask.dataframe as dd

storage_options={'account_name': ACCOUNT_NAME, 'account_key': ACCOUNT_KEY}

# Using abfs protocol
ddf = dd.read_csv('abfs://{CONTAINER}/{FOLDER}/*.csv', storage_options=storage_options)

# Using az protocol
ddf = dd.read_parquet('az://{CONTAINER}/folder.parquet', storage_options=storage_options)

You can install adlfs using pip or conda to enable az:// and abfs:// protocols in fsspec for interacting with Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2.

pip install adlfs
# or
conda install -c conda-forge adlfs

If no credentials are provided, adlfs defaults to using DefaultAzureCredential. You can explicitly provide credentials using the following keys in storage_options:

• connection_string: Azure connection string.
• account_name: The name of the storage account.
• account_key: The account access key.
• sas_token: Shared Access Signature token.
• tenant_id, client_id, and client_secret: Used together for Azure ServicePrincipal authentication.
• anon: (bool) Set to True for anonymous authentication. Defaults to False (checks AZURE_STORAGE_ANON env var first).
• location_mode: ("primary" or "secondary") Applies to RA-GRS accounts.

To read from a public storage blob, you must specify the 'account_name' in your storage_options. You may also need to set 'anon': True to use anonymous authentication.

# Example: Accessing NYC Taxi & Limousine Commission public data
storage_options = {'account_name': 'azureopendatastorage'}
ddf = dd.read_parquet('az://nyctlc/green/puYear=2019/puMonth=*/*.parquet', storage_options=storage_options)

When using AzureBlobFileSystem, use the storage account name (e.g., account_name="myaccount") rather than the full URL. The container name must be included as the first part of your path operations.

To list files in a container, pass the container name to fs.ls().

fs = adlfs.AzureBlobFileSystem(account_name="ai4edataeuwest")
# List files in the 'gbif' container
fs.ls("gbif")

When using write_bytes or write_text, you can pass keyword arguments directly into the underlying upload_blob method by using the ** unpacking syntax. This is useful for setting ContentSettings like content_type or content_encoding.

from azure.storage.blob import ContentSettings
fs = adlfs.AzureBlobFileSystem(account_name="ai4edataeuwest")

fs.write_bytes(
    path="path", 
    value=data, 
    overwrite=True, 
    **{"content_settings": ContentSettings(content_type="application/json", content_encoding="br")}
)

The adlfs.AzureBlobFile class represents an individual blob within Azure Blob Storage. It is typically returned when calling methods like fs.open() on an AzureBlobFileSystem instance and provides file-like interface methods (e.g., read(), write(), seek()) for interacting with the blob content.

from adlfs import AzureBlobFileSystem

fs = AzureBlobFileSystem(account_name='your_account_name')
with fs.open('container/path/to/blob.txt', 'rb') as f:
    data = f.read()

The adlfs.AzureBlobFileSystem class provides a filesystem interface for interacting with Azure Blob Storage. It implements the fsspec interface, allowing you to perform standard filesystem operations (like ls, cat, cp, mv, etc.) on Azure containers.

from adlfs import AzureBlobFileSystem

# Initialize the filesystem
fs = AzureBlobFileSystem(account_name='your_account_name', account_key='your_account_key') 
# Or using connection string/SAS tokens via constructor arguments

By default, write operations in AzureBlobFileSystem create BlockBlobs, which cannot be appended to. To create an AppendBlob, use mode="ab" when creating and operating on blobs.

Note: AppendBlobs are currently not available if hierarchical namespaces (ADLS Gen2) are enabled.

Instead of passing storage_options, you can set the following environment variables to authenticate:

• AZURE_STORAGE_CONNECTION_STRING
• AZURE_STORAGE_ACCOUNT_NAME
• AZURE_STORAGE_ACCOUNT_KEY
• AZURE_STORAGE_SAS_TOKEN
• AZURE_STORAGE_TENANT_ID
• AZURE_STORAGE_CLIENT_ID
• AZURE_STORAGE_CLIENT_SECRET

The adlfs package registers the following protocols with fsspec to map to specific filesystem implementations: