repository·develop·Indexed 25 days ago
https://github.com/piskvorky/smart_openA Python 3 library for the efficient streaming of very large files to and from various storage backends, including S3, GCS, Azure Blob Storage, HDFS, HTTP, and SFTP. It provides a unified API as a drop-in replacement for the built-in open() function, supporting transparent (de)compression for formats such as gzip, bz2, xz, lz4, and zstd.
profile_name parameter was removed in version 1.8.1. To use a specific AWS profile, you must now pass a boto3.Session object via transport_params.In version 8.0.0, several top-level wrappers were removed.
s3_iter_bucket: Import iter_bucket from smart_open.s3 instead.smart_open.smart_open(): Call smart_open.open() directly. If you were using ignore_extension=True, switch to compression='disable'.Once development dependencies are installed, you can execute the unit tests using pytest against the tests directory.
pytest testsgs:// still works as an alias, the canonical scheme for Google Cloud Storage is now gcs://.When working with versioned S3 buckets, you can retrieve a specific version of an object by passing version_id inside the transport_params dictionary. If version_id is not specified, smart_open reads the most recent version.
>>> import boto3
>>> from smart_open import open
>>> versions = ['KiQpZPsKI5Dm2oJZy_RzskTOtl2snjBg', 'N0GJcE3TQCKtkaS.gF.MUBZS85Gs3hzn']
>>> for v in versions:
... with open('s3://smart-open-versioned/demo.txt', transport_params={'version_id': v}) as fin:
... print(v, repr(fin.read()))
KiQpZPsKI5Dm2oJZy_RzskTOtl2snjBg 'second version\n'
N0GJcE3TQCKtkaS.gF.MUBZS85Gs3hzn 'first version\n'
>>> # If you don't specify a version, smart_open will read the most recent one
>>> with open('s3://smart-open-versioned/demo.txt') as fin:
... print(repr(fin.read()))
'second version\n'smart_open does not support zip files natively, but you can integrate it with Python's zipfile module. smart_open handles the I/O (streaming from/to local or remote storage) while zipfile handles the compression and member lookup.
>>> from smart_open import open
>>> import zipfile
>>> with open('sampledata/hello.zip', 'rb') as fin:
... with zipfile.ZipFile(fin) as zip:
... for info in zip.infolist():
... file_bytes = zip.read(info.filename)
... print('%r: %r' % (info.filename, file_bytes.decode('utf-8')))
'hello/': ''
'hello/en.txt': 'hello world!\n'
'hello/ru.txt': 'здравствуй, мир!\n'By default, smart_open does not install any dependencies to keep the installation size small. You can install specific storage solutions using extras defined in pyproject.toml.
To install specific transports:
pip install 'smart_open[s3,gcs,azure,http,webhdfs,ssh,zst,lz4]'To install all supported dependencies:
pip install 'smart_open[all]'Note on external dependencies:
kerberos=True) requires requests-kerberos: pip install requests-kerberosgss_* options) requires paramiko's GSSAPI support: pip install 'paramiko[gssapi]'hdfs:// and viewfs:// require the Hadoop hdfs command-line client to be installed and available on $PATH.pip install 'smart_open[s3,gcs,azure,http,webhdfs,ssh,zst,lz4]'smart_open via pip to enable efficient streaming of large files from various storage backends (S3, GCS, Azure, HDFS, HTTP, SFTP, etc.) in Python 3.By default, smart_open buffers the most recent 50MB part of a multipart upload in memory. To reduce memory footprint, you can:
min_part_size in transport_params (minimum 5MB).tempfile.NamedTemporaryFile object via the writebuffer key in transport_params. This reduces memory usage at the cost of additional disk I/O.# Option 1: Smaller part sizes
import boto3
from smart_open import open
tp = {"min_part_size": 5 * 1024**2}
with open("s3://bucket/key", "w", transport_params=tp) as fout:
fout.write(lots_of_data)
# Option 2: Using a temporary file buffer
import boto3
from smart_open import open
import tempfile
with tempfile.NamedTemporaryFile() as tmp:
transport_params = {"writebuffer": tmp}
with open("s3://bucket/key", "w", transport_params=transport_params) as fout:
fout.write(lots_of_data)smart_open.s3.iter_bucket no longer accepts individual AWS credentials as keyword arguments. You must now pass them via a single session_kwargs dictionary.Since version 3.0.0, smart_open does not install storage-specific dependencies (like boto3 for S3) by default to keep installations lean. You must install the extras for the protocols you intend to use.
pip install smart_open[s3]pip install smart_open[http]pip install smart_open[all]pip install smart_open[s3]
pip install smart_open[http]
pip install smart_open[all]To access public S3 buckets without using your own AWS credentials, pass a botocore.client.Config with signature_version=botocore.UNSIGNED via the transport_params dictionary.
>>> import boto3
>>> import botocore
>>> import botocore.client
>>> from smart_open import open
>>> config = botocore.client.Config(signature_version=botocore.UNSIGNED)
>>> params = {'client': boto3.client('s3', config=config)}
>>> with open('s3://commoncrawl/robots.txt', transport_params=params) as fin:
... fin.readline()
'User-Agent: *\n'