Apache OpenDAL Documentation

Apache OpenDAL is a unified, simple, efficient, reliable, and observable data access layer. It allows developers to seamlessly use different storage services through a single interface without needing to master individual service SDKs.

Key features include:

• Unified Interface: Access various storage services (e.g., local file system, HDFS, S3, OSS) using the same API.
• Common Operations: Supports standard data operations such as read, write, and list.
• Reliability & Optimization: Includes automatic retry mechanisms, request optimization, and observability.
• Performance: Designed for zero cost, mapping directly to API calls.
• Cross-language Support: Available via bindings for Rust, Python, Node.js, and C (in development).

The .NET binding provides a unified API to access over 50 storage services (like S3, GCS, Azure Blob, and the local filesystem) using a high-performance Rust core.

Key features include:

• Sync and async support: Every operation provides a blocking method and a Task-based ...Async counterpart with CancellationToken support.
• Stream wrappers: OpenReadStream and OpenWriteStream return standard .NET Stream objects.
• All services included: No build flags are required to enable specific backends; all are bundled.
• Composable layers: Support for retry, timeout, and concurrency limits.
• Presigned URLs: Async API support for services that allow URL presigning.

The Go binding provides a native interface to access various storage services (S3, GCS, Azure Blob, local filesystem, etc.) using the Rust core's performance. Key features include:

• Synchronous API: Every operation is a blocking method that returns a Go error. There are no async variants.
• Streaming: Supports streaming reads and writes via io.ReadSeekCloser and io.WriteCloser.
• Per-scheme packages: Each storage service is provided as a companion module under github.com/apache/opendal-go-services/<scheme> which must be imported alongside the main binding.
• Presigned URLs: For supported services, presigned URLs are returned as *net/http.Request objects.

The opendal Rust crate is the primary, most mature surface of the project. Key capabilities include:

• Async-first: Designed for asynchronous workflows, but also provides an optional blocking API.
• Streaming: Supports streaming reads and writes for handling data that exceeds available memory.
• Extensive Service Support: Provides a unified API for over 50 different storage services.
• Composable Layers: Allows adding functionality like retry, logging, timeout, and metrics via layers.
• Presigned URLs: Supports generating presigned URLs for compatible services (e.g., S3, GCS, Azblob).

The OpenDAL Java binding provides access to over 50 storage services (S3, GCS, Azure Blob, HDFS, local filesystem, etc.) with the following features:

• Sync and async modes: Use Operator for blocking calls or AsyncOperator for CompletableFuture-based asynchronous calls.
• Configuration options: Build operators using a typed ServiceConfig builder or a plain Map<String, String>.
• All-in-one binary: The native library bundles all backends; no additional build flags are required to enable specific services.
• Presigned URLs: Supported via the async API for compatible services.

Apache OpenDAL provides three primary modules for Spring integration:

• opendal-spring: The core integration module for general Spring applications.
• opendal-spring-boot-starter: A synchronous starter designed for Spring WebMVC applications.
• opendal-spring-boot-starter-reactive: An asynchronous starter designed for Spring WebFlux applications.

The Node.js binding provides several key features for interacting with storage services:

• Promise-based async API: Every operation returns a Promise that can be awaited.
• Synchronous variants: Each verb includes a *Sync form (e.g., readSync, writeSync) for blocking code execution.
• Streaming: Supports Node Readable and Writable streams for handling data that exceeds available memory.
• All services included: The package bundles all backends by default; no build flags are required.
• Composable layers: Supports adding layers for retry, timeout, logging, throttle, and concurrency limits.
• Presigned URLs: Supports generating presigned URLs for compatible services like S3, GCS, and Azblob.

The OpenDAL Python binding provides a native interface to access over 50 storage services (like S3, GCS, Azure Blob, HDFS, and the local filesystem) using a unified API with Rust-level performance.

Key features include:

• Sync and async support: Use Operator for synchronous operations or AsyncOperator for asynchronous operations using the same set of verbs.
• File-like objects: The op.open(path, "rb") method returns objects that support standard read/write/seek operations for streaming.
• All services included: Every backend is bundled in the wheel.
• Presigned URLs: Supports generating presigned URLs for supported services via the async API.

The OpenDAL Ruby binding provides a native interface to access over 50 storage services (like S3, GCS, Azure Blob, and the local filesystem) using a unified API.

Key features include:

• Synchronous API: Every operation runs on the calling thread; there is no asynchronous API.
• Unified API: The gem bundles all backends, which are selected via a scheme string during construction.
• File-like IO: The op.open(path, "r") method returns an object supporting standard read, write, seek, and readline operations for streaming.
• Middlewares: Supports OpenDAL layers for features like retry, concurrency limits, throttling, and timeouts.