repository·main·Indexed 43 days ago
https://github.com/apache/arrow-rs-object-storeA high-performance, asynchronous Rust library providing a generic interface for interacting with object storage backends, including AWS S3, Azure Blob Storage, Google Cloud Storage, local filesystems, memory, and HTTP/WebDAV. Version 0.14.2 supports advanced features like SSE-C, paginated listing, and custom HTTP clients via optional reqwest dependency.
The object_store crate is a high-performance, asynchronous, and idiomatic Rust library designed for interacting with various object storage backends. It allows developers to write code that can run across multiple cloud environments and local test setups by simply changing the runtime configuration.
Supported backends include:
The following backend enhancements were introduced in version 0.12.1:
0.4.0, the library introduced a builder-style configuration pattern for object stores. Instead of complex constructor arguments, you can use builder objects to incrementally configure your object store settings.The ObjectStore trait can be used as a trait object, allowing for polymorphic handling of different storage backends (e.g., S3, LocalFileSystem, InMemory) through a single interface.
impl ObjectStore for trait objectsStarting with version 0.13.0, many high-level convenience methods were moved out of the primary ObjectStore trait and into the ObjectStoreExt trait to keep the core trait lean. To use these methods, ensure you have imported the ObjectStoreExt trait into your scope.
Methods available in ObjectStoreExt:
getheadget_rangedeleteput_multipartuse object_store::ObjectStore;
use object_store::ObjectStoreExt; // Required to access convenience methods
// Now methods like .get() or .head() are accessed via ObjectStoreExt
let metadata = store.head(path).await?;object_store version 0.7.0, the GetResult type was updated to include range and ObjectMeta. This is a breaking change for any code relying on the previous structure of GetResult.Upgrading to v0.14.0 introduces several breaking changes that require updates to your configuration and dependency management:
cloud feature alias has been removed. You must now enable specific cloud provider features directly.reqwest: The reqwest dependency is now optional. If your application does not require HTTP-based object stores, you can disable it to reduce binary size and dependencies.*Result objects now include Extensions. Ensure your error handling logic accounts for this new field if you rely on metadata passed through results.ring to aws-rust-lc (via aws-lc-rs).# Example: If you previously used the 'cloud' feature,
# you must now specify your required providers explicitly.
# [object_store-0.14.0-dependencies]
# object_store = { version = "0.14.0", features = ["s3", "azure"] }The standard cloud features (aws, azure, gcp, and http) bundle a reqwest-based HTTP transport. If you need to target wasm32-wasip1 (where reqwest does not compile) or if you want to provide your own HTTP client, you must disable default features and use the corresponding *-base feature instead.
cargo build -p object_store --no-default-features --features aws-base --target wasm32-wasip10.11.1, object_store supports propagating environment variables as object store client options. This allows you to configure client behavior without changing code.Upgrading to version 0.13.0 introduces several breaking changes to the ObjectStore API. The core ObjectStore trait was refactored to separate core functionality from convenience methods, which were moved to the ObjectStoreExt trait.
Key Changes:
copy and copy_if_not_exists have been replaced by copy_opts. Similarly, rename and rename_if_not_exists have been replaced by rename_opts.ObjectStore trait to the ObjectStoreExt trait:getheadget_rangedeleteput_multipartdelete_stream now uses a 'static lifetime.// Example of the new pattern for consolidated options in v0.13.0
// Instead of: store.copy(src, dest)
// Use: store.copy_opts(src, dest, options)You can build object_store for the wasm32-unknown-unknown target. However, note that the cloud storage features (aws, azure, gcp, and http) are not supported on this target.
cargo build -p object_store --target wasm32-unknown-unknownobject_store version 0.6.0, the configuration pattern for ObjectStore was simplified. Users should adopt the new pattern for instantiating stores (e.g., S3, Azure, GCS) to ensure compatibility with the updated builder and options API.