DOCSEARCHbyAlgolia
Submit Docs

ARCO ERA5 Documentation

repository·main·Indexed 19 days ago

https://github.com/google-research/arco-era5

Recipes and curated datasets for Analysis-Ready and Cloud-Optimized (ARCO) ERA5 climate data. The project converts GRIB/NetCDF datasets into Zarr format to support high-performance cloud-based research and machine learning. It includes tools like weather-dl for data acquisition and weather-sp for splitting GRIB data by variable, as well as guidance on accessing 0.25° pressure, surface, and model level data via Google Cloud Storage.

Tokens
39.1K
Snippets
55
Records
68
Agent score
66%

What's inside ARCO ERA5

  1. Overview of ARCO ERA5 data types

    main

    The ARCO ERA5 project provides three distinct types of data stored in the gcp-public-data-arco-era5 bucket (located in the us-central1 region) to serve different research and machine learning needs:

    • Analysis Ready ($BUCKET/ar/): A unified, ML-ready version of both surface and atmospheric data in Zarr format. This version includes standard pre-processing and chunk optimization for common research workflows.
    • Cloud Optimized ($BUCKET/co/): A direct port of the gaussian-gridded ERA5 data to Zarr format, providing a cloud-optimized alternative to the original GRIB files without additional pre-processing.
    • Raw Data ($BUCKET/raw/): The original raw GRIB and NetCDF data files.

    Data is updated on a monthly cadence (around the 9th of each month) with a 3-month delay for stable ERA5. Preliminary ERA5T data is available with approximately a 1-week delay.

  2. Understand the limitations and cautions of ARCO-ERA5 data

    main

    When using ARCO-ERA5 reanalysis data, be aware of the following limitations to avoid incorrect scientific conclusions:

    • Coastal Locations: ERA5 uses a land-surface model that blends water and land influence based on a land fraction. Near coastlines, sharp changes in land fraction between neighboring grid cells can cause artificial variations in daily temperature ranges.
    • Complex Topography: The dataset may struggle in areas with large topographic variations (e.g., high mountain ranges like the Everest region). ERA5 topography may miss high peaks and valley structures, which can lead to inaccuracies in temperature and precipitation patterns.
    • Precipitation Accuracy: Precipitation variables are not directly constrained by observations. It is strongly recommended to validate ERA5 precipitation against observed data.
    • Nature of Reanalysis: Remember that reanalysis is an estimate of past weather, not an error-free observation.

  3. Identify available data time ranges using Zarr metadata

    main

    To find the most recent data available in a Zarr store, examine its metadata attributes. These attributes define the temporal coverage of the dataset in UTC. Note that both start and end times are inclusive.

    For standard ERA5, use:

    • valid_time_start: The start date of the data.
    • valid_time_stop: The end date of the data.
    • last_updated: The timestamp of the most recent update.

    For ERA5T (preliminary version), use:

    • valid_time_stop_era5t instead of valid_time_stop to determine the end date.

  4. Visualize Metview fieldsets

    main

    Metview uses a hierarchical plotting system based on Views (e.g., geoview for horizontal maps, thermoview for thermodynamic diagrams).

    Key components of a plot include:

    • mcont: Defines the contour/shaded plot style.
    • mcoast: Controls coastline, political boundaries, and land/sea shading.
    • mtext: Controls text and titles.
    • mwind: Controls vector wind plots.

    To plot a horizontal map with specific boundaries, use mapCorners (a list of [lat_min, lon_min, lat_max, lon_max]) or mapAreaNames.

    import metview as mv

# Example: Plotting geopotential height on a lat-lon grid
# ll_fieldset is a fieldset interpolated to a regular grid
ca_corners = [50, -130, 30, -110]

gh_contour_plot(ll_fieldset,
                 mapCorners=ca_corners,
                 mapAdministrativeBoundaries="on",
                 mapAdministrativeBoundariesCountriesList=["USA","CAN"],
                 level=137)

  5. Reproduce Cloud-Optimization using Apache Beam and Dataflow

    main

    The project uses Apache Beam to ensure portability across cloud runners. To reproduce the dataset using GCP's Dataflow, follow these steps:

    1. Prerequisites: Ensure you have a GCP project with GCS read/write access and full Dataflow permissions.
    2. Environment Setup: Export the following environment variables:
      • PROJECT: Your GCP project ID.
      • REGION: The target region (e.g., us-central1).
      • BUCKET: Your Beam runner bucket.
    3. Execution: Run the provided recipes. You can view the documentation for specific scripts using pydoc or check available command-line options with the --help flag.

    Example of checking help for a script:

    python src/model-levels-to-zarr.py --help
    export PROJECT=<your-gcp-project>
export REGION=us-central1
export BUCKET=<your-beam-runner-bucket>

python src/model-levels-to-zarr.py --help

  6. Convert Divergence and Vorticity to U/V Winds

    main

    Since ERA5 model-level wind variables are stored as spherical harmonics, they are represented as divergence (d) and vorticity (vo) rather than vector components. Use metview.uvwind to perform the conversion in spectral space.

    Workflow:

    1. Convert d and vo to u and v components using mv.uvwind (set truncation=639 for ERA5).
    2. Interpolate the resulting spectral field to a reduced Gaussian grid (N320).
    3. Interpolate from the Gaussian grid to a regular lat-lon grid.
    import metview as mv

# 1. Spectral conversion
uv_wind_spectral = mv.uvwind(data=wind_fieldset, truncation=639)

# 2. Interpolate to Gaussian grid
uv_wind_gg = mv.read(data=uv_wind_spectral, grid='N320')

# 3. Interpolate to regular lat-lon grid
uv_wind = mv.read(data=uv_wind_gg, grid=[0.25, 0.25])

  7. Authenticate with Google Cloud Platform

    main

    To access the ERA5 data stored on Google Cloud Storage, you must authenticate using Application Default Credentials (ADC). This allows programmatic access to GCP resources.

    Run the following command in your terminal:

    gcloud auth application-default login

    Note: Do not use gcloud auth login, as that is intended for interactive user authentication and may not provide the necessary credentials for data libraries to access GCP resources programmatically.

    gcloud auth application-default login

  8. Split GRIB data by variable using weather-sp

    main

    Because GRIB files can contain heterogeneous data (multiple levels or grids) that Pangeo Forge Recipes cannot yet handle, you must split certain datasets (soil and pcp) by variable using the weather-sp tool.

    Prerequisites

    1. Install google-weather-tools (version 0.3.0 or higher):
      pip install google-weather-tools>=0.3.0
    2. Ensure you have read access to the source datasets in cloud storage.

    Execution

    1. Dry Run: Preview the split using --input-pattern and --output-template:
      export DATASET=soil
weather-sp --input-pattern "gs://gcp-public-data-arco-era5/raw/ERA5GRIB/HRES/Month/**/*_hres_$DATASET.grb2" \
  --output-template "gs://gcp-public-data-arco-era5/raw/ERA5GRIB/HRES/Month/{1}/{0}.grb2_{typeOfLevel}_{shortName}.grib" \
  --dry-run
    2. Execute on Dataflow: Run the split job on a Beam runner:
      export DATASET=soil
export PROJECT=<your-project>
export BUCKET=<your-bucket>
export REGION=us-central1

weather-sp --input-pattern "gs://gcp-public-data-arco-era5/raw/ERA5GRIB/HRES/Month/**/*_hres_$DATASET.grb2" \
  --output-template "gs://gcp-public-data-arco-era5/raw/ERA5GRIB/HRES/Month/{1}/{0}.grb2_{typeOfLevel}_{shortName}.grib" \
  --runner DataflowRunner \
  --project $PROJECT \
  --region $REGION \
  --temp_location gs://$BUCKET/tmp \
  --disk_size_gb 100 \
  --job_name split-soil-data
    3. Repeat for pcp: Set export DATASET=pcp and run the same commands to process precipitation data.
    export DATASET=soil
export PROJECT=<your-project>
export BUCKET=<your-bucket>
export REGION=us-central1

weather-sp --input-pattern "gs://gcp-public-data-arco-era5/raw/ERA5GRIB/HRES/Month/**/*_hres_$DATASET.grb2" \
  --output-template "gs://gcp-public-data-arco-era5/raw/ERA5GRIB/HRES/Month/{1}/{0}.grb2_{typeOfLevel}_{shortName}.grib" \
  --runner DataflowRunner \
  --project $PROJECT \
  --region $REGION \
  --temp_location gs://$BUCKET/tmp \
  --disk_size_gb 100 \
  --job_name split-soil-data

  10. Validate ERA5 data consistency

    main

    The script raw/gcs_data_consistency_checker.py validates files within the gcp-public-data-arco-era5 Google Cloud Storage bucket. It checks that required files for a specific year range exist and identifies any extra files present in the bucket.

    Configuration

    Modify the following variables within the script:

    • BUCKET: The name of the GCS bucket containing the data.
    • START_YEAR: The beginning of the validation range.
    • END_YEAR: The end of the validation range.

    Prerequisites

    • Python 3.x
    • Python packages: fsspec, pandas
    python raw/gcs_data_consistency_checker.py

  11. Check data licensing and usage terms

    main

    ARCO-ERA5 data can be used for both research and commercial purposes, provided you adhere to the Copernicus license.

    • Researchers: Should cite the ARCO-ERA5 presentation and the original ERA5 dataset (see 'How to cite this work' for details).
    • Commercial Users: Must provide acknowledgement to the Copernicus Climate Change Service according to the Copernicus Licence terms.

  12. Explore ERA5 with Colab Notebooks

    main

    You can interactively explore the ERA5 datasets using provided Google Colab notebooks:

    • Surface Reanalysis Walkthrough: Learn how to work with surface-level ERA5 data.
    • Model Levels Walkthrough: Explore atmospheric model levels data.
    https://github.com/google-research/arco-era5/blob/main/docs/0-Surface-Reanalysis-Walkthrough.ipynb
https://github.com/google-research/arco-era5/blob/main/docs/1-Model-Levels-Walkthrough.ipynb