DOCSEARCHbyAlgolia
Submit Docs

gcsfs Documentation

repository·main·Indexed 18 days ago

https://github.com/fsspec/gcsfs

A Python library providing a file-system-like interface to Google Cloud Storage (GCS), built on top of fsspec. It allows users to interact with cloud buckets as if they were local directories. The documentation includes detailed guides on automating performance benchmarks and macrobenchmarks using Google Cloud Build, GKE, and BigQuery for regression detection and historical analysis.

Tokens
26.6K
Snippets
64
Records
117
Agent score
63%

What's inside gcsfs

  1. Overview of GCSFS Subsystem Benchmarks Automation

    main

    This automation suite uses Google Cloud Build to run GCSFS subsystem benchmarks on ephemeral Compute Engine VMs. It is designed for historical performance analysis by automating the entire lifecycle: provisioning infrastructure, running benchmark groups, uploading results to Cloud Storage, and ingesting data into BigQuery.

    There are two primary pipelines:

    1. Run pipeline (subsystembenchmarks-cloudbuild.yaml): Provisions a VM, executes the selected benchmark group, uploads JSON/CSV artifacts, and cleans up resources.
    2. Ingestion pipeline (subsystembenchmarks-ingestion-cloudbuild.yaml): Processes uploaded CSVs from a staging table and appends new results to a partitioned BigQuery history table.

  2. Overview of GCSFS Macrobenchmarks Automation

    main

    The GCSFS Macrobenchmarks Automation provides a Cloud Build-driven pipeline to run end-to-end performance benchmarks for gcsfs.

    It consists of two distinct pipelines:

    1. Run pipeline (macrobenchmarks-cloudbuild.yaml): Provisions ephemeral GKE infrastructure, runs a Llama 3.1 8B PyTorch-Lightning CPU simulation as a Kubernetes JobSet, scrapes metrics from Cloud Logging and Cloud Monitoring into a summary CSV, and uploads it to a results bucket.
    2. Ingestion pipeline (macrobenchmarks-ingestion-cloudbuild.yaml): Loads the summary CSVs from the results bucket into BigQuery for historical analysis.

    Note: Running these pipelines bills real compute and storage in your own GCP project. The default configuration uses high-resource machines (c4-standard-192) and can run for up to 6 hours.

  3. Understand the GCSFS Benchmarks Automation pipelines

    main

    GCSFS Benchmarks Automation uses Google Cloud Build to automate two distinct pipelines: running performance benchmarks and ingesting the results into BigQuery. This allows for nightly performance tracking and regression detection across different bucket types like Regional, Zonal, and HNS (Hierarchical Namespace).

    1. Benchmarks Run Pipeline

    Defined in benchmarks-cloudbuild.yaml, this pipeline is ephemeral. It:

    • Sets up infrastructure (SSH keys, temporary GCS buckets, and a high-performance Compute Engine VM).
    • Executes the benchmark suite (run.py) on the VM.
    • Uploads CSV results and JSON logs to a persistent GCS Results Bucket at gs://<results_bucket>/<date>/<run_id>/.
    • Cleans up all temporary resources (VM, buckets, and keys).

    2. Benchmarks Ingestion Pipeline

    Defined in benchmarks-ingestion-cloudbuild.yaml, this pipeline moves data from GCS to BigQuery. It:

    • Uses an External Table (staging) to point to CSV files in GCS using a schema from benchmarks_schema.json.
    • Uses a partitioned history table for long-term storage.
    • Handles schema evolution by automatically adding new columns detected in the staging table to the history table.
    • Ensures idempotency by using the source_uri to prevent duplicate entries.

  4. Overview of Prefetching components

    main

    The prefetching architecture consists of four decoupled components:

    • RunningAverageTracker: Monitors recent read request byte sizes and calculates a rolling average to define the base I/O size.
    • PrefetchProducer: An asyncio background task that calculates the prefetch_size and pushes concurrent download promises into a shared queue.
    • PrefetchConsumer: Manages the local memory buffer, consumes background tasks from the queue, assembles byte strings, and slices the specific data requested by the user.
    • BackgroundPrefetcher: The main orchestrator that ties the producer and consumer together, manages file seek operations (soft vs. hard seeks), and handles resource cleanup (flushing sockets and buffers) upon closing.

  5. How GCSFS Adaptive Concurrent Prefetching works

    main

    The GCSFS prefetcher is an adaptive engine designed to balance aggressive background data fetching with efficient memory management. It monitors application reading patterns to decide whether to prefetch data or conserve resources.

    It operates based on three primary behavioral states:

    1. Sequential Reading: When the engine detects a consistent streak of sequential reads, it ramps up background fetching to maintain a buffer ahead of the application. This ensures the user does not wait on network latency.
    2. Random Seeks: If the application performs random seeks (jumping around the file), the engine detects the broken sequential streak and immediately drops the background buffer to zero. This prevents wasting network bandwidth and memory on data that is unlikely to be used.
    3. Dynamic Scaling: The engine calculates a rolling average of read sizes. If the application transitions from small sequential reads to larger sequential reads, the prefetcher scales its background buffer to match the new workload intensity.

  6. How bucket type detection and caching works

    main

    To determine if a bucket is HNS-enabled or Zonal, ExtendedGcsFileSystem uses the Cloud Storage Control API's get_storage_layout method.

    • Caching: Once the bucket type is identified, the filesystem caches the layout to prevent repeated API overhead.
    • Error Handling: If detection returns UNKNOWN (e.g., due to network issues), the result is not cached. This prevents transient failures from permanently disabling HNS optimizations.
    • Fallback: If the type is UNKNOWN, the filesystem gracefully falls back to standard flat-namespace operations and will retry the lookup on subsequent requests.

  7. Understand limitations and behaviors of Rapid Storage (Rapid Buckets) in gcsfs

    main

    When using Rapid Storage with gcsfs, several behaviors differ from standard flat GCS buckets:

    • HNS Requirement: Rapid Storage requires a Hierarchical Namespace (HNS) to be enabled. All HNS directory semantics (like real folder resources and strict mkdir behavior) apply.
    • Native Appends: Unlike standard buckets where appending requires a full rewrite, Rapid Storage supports native appends. Opening a file in append mode (ab) allows the object size to grow in real-time.
    • Single Writer Constraint: Only one active writer can exist for an appendable object at a time. Establishing a new write stream will interrupt the original stream and trigger a Cloud Storage error.
    • Finalization: Once an object is finalized (e.g., the stream is closed), it can no longer be appended to. gcsfs keeps objects unfinalized by default to support appends. Note that the autocommit argument is not supported for Rapid buckets.
    • Metadata Restrictions: You cannot set contentType, metadata, fixed_key_metadata, or kmsKeyName during upload to Rapid buckets. Additionally, metadata/object size may be stale for unfinalized objects being appended to.
    • Feature Incompatibilities:
      • The transaction feature is not supported (it relies on discard(), which is unsupported).
      • Standard GCS features may be incompatible; check the official Google Cloud documentation for the full list.
    • Write Buffering: The flush interval is 16 MiB (compared to 5 MiB in regional buckets). Calling flush is more expensive because the gRPC stream must update the persisted_size.

  8. Retry behavior for Rapid Storage (Zonal Buckets)

    main

    Zonal buckets (ZonalFile) use specialized gRPC clients. Control plane operations use the same mechanism as HNS buckets. Data plane operations (reads/writes) use the underlying Google Cloud Python SDK's gRPC retry behavior.

    AsyncMultiRangeDownloader (MRD) - Reads

    Applicable Methods: open, download_ranges.

    • Retriable Errors: InternalServerError, ServiceUnavailable, DeadlineExceeded, TooManyRequests (429), and Aborted (allows resuming from last successful byte offset).
    • Backoff: Exponential; initial=1.0s, maximum=60.0s, multiplier=2.0.
    • Overall Deadline: 120.0s.

    AsyncAppendableObjectWriter (AAOW) - Writes

    Applicable Methods: open, append.

    • Note: flush and finalize do not have automatic retry logic.
    • Retriable Errors: InternalServerError, ServiceUnavailable, DeadlineExceeded, TooManyRequests (429), and BidiWriteObjectRedirectedError (handled by re-opening stream and resuming from last persisted offset).
    • Backoff: Exponential; initial=1.0s, maximum=60.0s, multiplier=2.0.
    • Overall Deadline: 120.0s.

  9. How GCSFS Adaptive Concurrent Prefetching works

    main

    The prefetching system uses an asynchronous pipeline to hide network latency by predicting and downloading data before the application requests it. Unlike the Linux kernel which uses fixed 4KB pages and exponential window scaling, GCSFS adapts to cloud environments using these principles:

    • Dynamic I/O Sizing: Uses a RunningAverageTracker to monitor recent read request sizes and sets the base block size to match the user's actual workload (e.g., 100MB instead of 4KB).
    • Immediate Triggering: Prefetching is triggered immediately upon the consumption of the first block in a sequence to prevent stalling on TCP/TLS handshakes.
    • Linear Scaling: To avoid excessive egress costs and network congestion (the "noisy neighbor" problem), the prefetch window scales linearly (sequential_streak * io_size) rather than exponentially.
    • Software Multiplexing: Since single HTTP streams are bandwidth-capped, a PrefetchProducer calculates a split_factor to multiplex the prefetch window into multiple concurrent HTTP Range requests using asyncio.

  10. How adaptive prefetching works with GCSFile

    main

    The prefetcher is integrated into the GCSFile class and replaces standard sequential fetching when enabled.

    Lifecycle and Interaction:

    • Initialization: During GCSFile.__init__, if enabled, a BackgroundPrefetcher is instantiated and attached to self._prefetch_engine.
    • Fetching: GCSFile._async_fetch_range is mapped to the prefetcher. When you call file.read(size), the request is delegated to self._prefetch_engine._fetch(start, end).
    • Concurrency: The prefetcher returns requested bytes from its local queue while a background producer continues pulling chunks from GCS.
    • Cleanup: Calling file.close() triggers _prefetch_engine.close(), which cancels pending network tasks and clears memory buffers to prevent leaks.

  11. Manage test buckets when using real GCS

    main

    When testing against a real GCS endpoint, the behavior regarding bucket lifecycle depends on whether the bucket names you provide already exist:

    • Existing Buckets: If you provide names of buckets that already exist, the test suite will manage objects within them. Warning: The test suite will clear the contents of the bucket at the beginning and end of the test run. Do not use buckets containing important data.
    • New Buckets: If you provide names of buckets that do not exist, the test suite will create them for the duration of the test run and automatically delete them during the final cleanup.

  12. Default retry behavior for standard GCS buckets

    main

    For standard buckets, gcsfs uses a custom retry_request decorator to handle transient errors. Most high-level operations benefit from this automatically.

    Applicable Methods:

    • ls / _ls: Listing objects and prefixes.
    • info / _info: Retrieving object metadata.
    • cat / _cat_file: Reading object contents.
    • get / _get_file: Downloading objects.
    • put / _put_file: Uploading objects (including resumable uploads).
    • mkdir / _mkdir: Creating buckets.
    • rm / _rm_file: Deleting objects.
    • mv / _mv_file: Moving/renaming objects.
    • cp / _cp_file: Copying objects.

    Retry Configuration:

    • Number of Retries: Default is 6 (defined by GCSFileSystem.retries).
    • Backoff Strategy: Exponential backoff with jitter: min(random.random() + 2 ** (retry - 1), 32).
    • Timeouts: Individual requests use requests_timeout (if configured in GCSFileSystem.__init__). There is no total deadline for the retry loop; it will attempt up to 6 retries regardless of total elapsed time.

    Retriable Errors:

    • requests.exceptions.ChunkedEncodingError, ConnectionError, ReadTimeout, Timeout, ProxyError, SSLError, ContentDecodingError.
    • google.auth.exceptions.RefreshError.
    • aiohttp.client_exceptions.ClientError.
    • ChecksumError.
    • HTTP status codes: 500-504, 408, 429.
    • HTTP 401 with "Invalid Credentials" message (auth expiration).