DOCSEARCHbyAlgolia
Submit Docs

Google Cloud Storage Python Client Library

repository·main·Indexed 19 days ago

https://github.com/googleapis/python-storage

A Python client library for interacting with Google Cloud Storage, enabling the management of buckets and objects (unstructured data). The library supports operations such as creating buckets with various configurations, managing HMAC keys, configuring CORS, handling IAM bindings, and performing file operations including downloads, uploads, and composition.

Tokens
27.2K
Snippets
118
Records
134
Agent score
68%

What's inside googleapis-python-storage

  1. Use the Google Cloud Client Library for Python

    main

    The samples in this repository are built using the Google Cloud Client Library for Python. For detailed API usage, documentation, or to report issues, refer to the official resources.

  2. Understand the StorageTransport inheritance structure

    main

    The StorageTransport Abstract Base Class (ABC) serves as the foundation for all transport implementations in the library. Depending on your requirements for protocol (gRPC vs. REST) and execution model (Synchronous vs. Asynchronous), you will interact with different child classes:

    • gRPC (Synchronous): Use StorageGrpcTransport (defined in grpc.py).
    • gRPC (Asynchronous): Use StorageGrpcAsyncIOTransport (defined in grpc_asyncio.py).
    • REST (Synchronous): Use StorageRestTransport (defined in rest.py). This class uses METHOD inner classes derived from the private _BaseMETHOD classes in _BaseStorageRestTransport.

    Note that _BaseStorageRestTransport and its inner class _BaseMETHOD are private implementation details and should not be used directly by end-users.

  3. How ETag, Generation, and Metageneration work for preconditions

    main

    Cloud Storage uses preconditions to ensure safe read-modify-write updates and conditional operations. By providing specific identifiers, you can instruct the service to only perform a request if the object is in the expected state.

    Core Concepts

    • ETag: A unique identifier returned in response headers and resource bodies. It changes whenever the underlying data changes. The ETag attribute is read-only.
    • Generation: A unique number assigned to a specific version of a blob. When you upload a new version of a file, the generation changes and the metageneration is reset to 1. The generation attribute is read-only.
    • Metageneration: A version number for the resource's metadata.
      • For a Bucket, it is initialized to 1 upon creation.
      • For a Blob, it is initialized to 1 upon the first upload.
      • Every time the bucket or blob's metadata is patched or updated, the metageneration increments. The metageneration attribute is read-only.

  4. Thread safety and multiprocessing best practices

    main

    The Google Cloud Storage Python client uses the requests library by default, making it safe to share client instances across multiple threads.

    However, when using multiprocessing, you should avoid sharing a client instance created in a parent process. Instead, create new client instances within the child process after multiprocessing.Pool or multiprocessing.Process has invoked os.fork() to ensure stability and avoid resource conflicts.

  5. Understand IAM vs ACLs in Cloud Storage

    main

    Cloud Storage provides two parallel access control systems:

    1. ACLs (Access Control Lists): Best for fine-grained, individual object-level control.
    2. IAM (Identity and Access Management): Best for project-level or bucket-level control. IAM is recommended for permissions applying to multiple objects to reduce exposure risk.

    Note: You can enable uniform bucket-level access to disable ACLs entirely and use IAM exclusively.

  6. Understand checksum defaults in Python Storage 3.0

    main

    Starting with version 3.0, uploads and downloads use an "auto" checksum policy.

    • Behavior: "Auto" uses crc32c checksums by default. If the fast C extension for crc32c is unavailable, it falls back to md5.
    • Ranged Downloads: Downloads with start or end parameters still do not support checksumming.
    • Important Note for Blob.upload_from_file(): This method now requires the file to be opened in bytes mode. Passing a file in string mode will now raise a TypeError due to the new checksum defaults.

  7. Enable OpenTelemetry tracing for Cloud Storage

    main

    This is a PREVIEW FEATURE. You can use OpenTelemetry to generate traces for Cloud Storage calls.

    1. Install the tracing extra:
    pip install google-cloud-storage[tracing]
    1. Enable tracing via environment variable:
    export ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES=True
    1. Configure an exporter (e.g., Google Cloud Trace) and instrument the requests library to trace underlying HTTP calls.
    pip install google-cloud-storage[tracing]
export ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES=True

  8. Set up authentication for Google Cloud Storage

    main

    To use the Google Cloud Storage Python client, you must configure authentication for your application. This typically involves setting up credentials that allow your code to interact with Google Cloud services.

    For detailed instructions on setting up credentials for your specific environment (such as using Service Accounts or Application Default Credentials), refer to the official Google Cloud Authentication Getting Started Guide.

    https://cloud.google.com/docs/authentication/getting-started

  9. Configure retry policies for Google Cloud Storage

    main

    The library provides default retry policies based on the idempotency of the API request. You can override these defaults to customize how transient errors are handled.

    Default Retry Behaviors

    • Always idempotent: Uses DEFAULT_RETRY (retries any transient error).
    • Idempotent if generation matches: Uses DEFAULT_RETRY_IF_GENERATION_SPECIFIED (requires generation or ifGenerationMatch header).
    • Idempotent if metageneration matches: Uses DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED (requires ifMetagenerationMatch header).
    • Idempotent if ETAG matches: Uses DEFAULT_RETRY_IF_ETAG_IN_JSON (requires ETAG in payload).
    • Never idempotent: Retries are suppressed by default (retry=None).

    Customizing Retries

    You can customize retries using several methods:

    1. Disable retries: Pass retry=None.
    2. Modify default policy: Use the .with_XXX methods on DEFAULT_RETRY to adjust timeout or delay parameters.
    3. Custom google.api_core.retry.Retry instance: Define specific retriable exceptions and backoff logic.
    4. ConditionalRetryPolicy: Wrap a retry policy to activate it only when specific conditions (like an ETAG being present) are met.
    from google.cloud.storage.retry import DEFAULT_RETRY

# Customize retry with a timeout of 500 seconds (default=120 seconds).
modified_retry = DEFAULT_RETRY.with_timeout(500.0)

# Customize delay parameters: initial wait, multiplier, and maximum wait time.
# Defaults: initial=1.0, multiplier=2.0, maximum=60.0
modified_retry = modified_retry.with_delay(initial=1.5, multiplier=1.2, maximum=45.0)

  10. Run Google Cloud Storage Python Samples

    main

    The samples/snippets directory contains standalone Python scripts demonstrating various Google Cloud Storage operations. Most samples require specific command-line arguments such as <BUCKET_NAME>, <BLOB_NAME>, or <PROJECT_ID> to function. You can run these scripts directly using the Python interpreter.

    python <sample_name>.py <ARGUMENTS>

  12. Set up a development environment

    main

    To contribute to python-storage, you must first fork the repository on GitHub and clone your fork locally. You should also configure an upstream remote pointing to the official googleapis/python-storage repository to keep your local version synchronized with changes from the main project.

    Follow these steps to initialize your environment:

    1. Fork the repository on GitHub.
    2. Clone your fork to a local directory (e.g., hack-on-python-storage).
    3. Add the upstream remote.
    4. Fetch and merge the latest changes from the official main branch.
    $ cd ${HOME}
$ git clone git@github.com:USERNAME/python-storage.git hack-on-python-storage
$ cd hack-on-python-storage
# Configure remotes to pull changes from the official repository
$ git remote add upstream git@github.com:googleapis/python-storage.git
# Fetch and merge changes from upstream into main
$ git fetch upstream
$ git merge upstream/main