repository·main·Indexed 20 days ago
https://github.com/developmentseed/obstoreA high-performance Python library powered by Rust providing a unified, high-throughput interface to Amazon S3, Google Cloud Storage, Azure Blob Storage, and S3-compliant APIs like Cloudflare R2. It supports sync and async APIs, streaming operations, and zero-copy data exchange via the buffer protocol. Version 0.11.0.
pyo3-bytes provides an integration between the Rust bytes crate and pyo3. It introduces the PyBytes type, which is a wrapper around bytes::Bytes that supports the Python buffer protocol. This allows for efficient data transfer between Rust and Python by avoiding unnecessary copies.obstore is a high-throughput Python interface to S3-compliant APIs (Amazon S3, Google Cloud Storage, Azure Storage, etc.) powered by Rust. Key features include:
fsspec integration.dicts).Beyond performance, Obstore provides several features designed to simplify object storage workflows:
list calls handle pagination automatically behind the scenes.Obstore provides two distinct API designs for performing get operations: a Method API and a Functional API.
store.get(...)). Use this when you have an existing ObjectStore instance and want to use its encapsulated state.obstore.get(store, ...)). Use these for a more functional programming style or when you prefer passing the store as an explicit argument.Obstore's performance impact depends on the type of operation being performed. Use the following mental model to choose your implementation strategy:
obstore.put uses multipart uploads by default, which may provide efficiency gains by uploading various parts of a file concurrently.To avoid the overhead of creating many small Python dictionaries when listing large buckets, use return_arrow=True in the list method. This returns each chunk of results as an Arrow RecordBatch.
This approach allows for zero-copy conversion to other Arrow-backed libraries. This requires the arro3-core dependency.
Supported zero-copy conversions:
pyarrow.record_batch(record_batch)polars.DataFrame(record_batch)pyarrow.record_batch(record_batch).to_pandas(types_mapper=pd.ArrowDtype)arro3.core.RecordBatch(record_batch)import pandas as pd
import pyarrow as pa
from obstore.store import S3Store
store = S3Store("sentinel-cogs", region="us-west-2", skip_signature=True)
stream = store.list(chunk_size=20, return_arrow=True)
for record_batch in stream:
# Convert to pyarrow (zero-copy), then to pandas
df = pa.record_batch(record_batch).to_pandas()
print(df.iloc[:5].to_markdown(index=False))
breakThe pyo3-bytes interface is not 100% safe because the Python buffer protocol does not enforce buffer immutability.
Warning: Python users must be instructed not to mutate buffers that have been passed to Rust to avoid undefined behavior.
Native authentication refers to authentication methods supported directly by the underlying Rust object_store library. Using native authentication is the most efficient method because obstore does not need to call into Python to update or manage credentials.
Native authentication credentials are applied in the following priority order:
config parameter or as keyword arguments during store construction.obstore looks for corresponding environment variables.Explicitly passed parameters will always override values found in environment variables.
Obstore uses a functional API design. Instead of calling methods directly on a store instance (e.g., store.put(...)), you must pass the store instance as the first argument to top-level functions provided by the obstore module (e.g., obstore.put(store, ...)).
This design ensures that generic operations work across all types of stores and middlewares (like prefixes) without needing to re-implement or re-expose specific methods on every wrapper.
import obstore as obs
from obstore.store import AzureStore
store = AzureStore()
# Correct: Use the top-level function
obs.put(store, ...)
# Incorrect: Do not attempt to call methods on the store instance
# store.put(...) The functional API provides top-level functions in the obstore module to perform operations. Unlike the method-based API, you must explicitly pass the store instance as the first argument to every function call. This API is required for certain features that are not universally supported by all object storage backends.
import obstore as obs
from obstore.store import S3Store
store = S3Store("bucket", ...)
buffer = obs.get_range(store, "path", start=0, end=16384)If you want your library to accept ObjectStore instances created by other Python libraries (such as obstore), use AnyObjectStore as your function parameter. AnyObjectStore acts as a wrapper that can handle both PyObjectStore (from your own library) and PyExternalObjectStore (from external libraries).
Warning on ABI Stability: Because object_store does not have a stable ABI, using AnyObjectStore to accept external stores results in the store being recreated. This means connection pooling cannot be shared between your library and the external library.
Obstore supports Python's pickle protocol, which is useful for distributed execution frameworks like Dask that need to manage and share store state across different workers.
Important Limitations:
obstore versions).MemoryStore because the raw state of the store cannot be accessed for serialization.