AIStore (AIS) Documentation

AIStore (AIS) is a highly available, partition-tolerant distributed storage system designed for petascale deep learning workloads. It provides read-after-write consistency and supports data protection through n-way mirroring and erasure coding.

Beyond standard storage, AIS functions as an ETL (Extract-Transform-Load) engine that can shuffle datasets and run custom workloads. It is designed to be lightweight and can be deployed on a single Linux machine or a bare-metal cluster, with or without Kubernetes.

Key Features:

• Direct I/O: Supports direct (compute <=> disk) I/O flows, avoiding the latency overhead of breaking objects into uniform fragments with separate metadata manifests.
• Data Sharding: Focuses on (re)sharding datasets prior to workloads to ensure optimal consumption by training applications.
• Linear Scalability: Achieves linear scalability under random-read workloads, where total-throughput = N * single-disk-throughput (N being the total number of clustered disks).
• APIs: Provides both S3 and native APIs.
• Extensibility: Can act as fast storage or an on-demand cache in front of supported backends, including Cloud buckets.

AIStore is a lightweight, distributed storage stack designed for AI workloads. Key capabilities include:

• Multi-Cloud Access: Manage content across AWS S3, GCS, Azure, and OCI with namespace-aware bucket identity.
• High Availability: Features redundant control/data planes, self-healing, n-way mirroring, and erasure coding.
• API Support: Provides a native HTTP-based API (with Go and Python SDKs) and a compliant Amazon S3 API for unmodified S3 clients.
• Chunked Objects: High-performance object representation supporting parallel reads, rechunking, and checksum-protected manifests.
• ETL Offload: Execute data transformations inline (on-the-fly) or offline (batch processing) close to the data.
• Get-Batch: A specialized operation for ML/AI pipelines that retrieves multiple objects/archives in a single call, assembling them into a specific serialization format (like TAR) in the requested order.
• Unified Namespace: Connect independent clusters to provide unified access to datasets across the organization.

The Batch Module in the AIStore Python SDK allows you to load multiple objects in a single request. Instead of multiple individual GET requests, AIStore aggregates the requested objects (or sharded files) into a single ordered archive (e.g., TAR, ZIP) and streams it back. This reduces network round-trips and connection overhead, which is particularly beneficial for high-performance ML training workloads.

Key features include:

• Multi-Bucket Support: A single batch can span multiple local or remote cloud buckets (S3, GCS, Azure, OCI).
• Strict Ordering: The archive contains objects in the exact sequence they were requested.
• Archive Path Support: You can extract specific files from within archive objects (like WebDataset) without downloading the entire archive.
• Byte Range Reads: Request specific byte ranges from objects or extracted archive files.

from aistore.sdk import Client

client = Client("http://localhost:8080")
bucket = client.bucket("example")

# Batch get using object names as strings
object_names = ["obj1", "obj2", "dir/file3.jpg"]
batch = client.batch(object_names, bucket=bucket)

for obj_info, data in batch.get():
    print(f"Object: {obj_info.obj_name}, Size: {len(data)}")