DOCSEARCHbyAlgolia
Submit Docs

WeatherBench 2 Documentation

repository·main·Indexed 20 days ago

https://github.com/google-research/weatherbench2

A benchmarking framework for global weather forecasting models that provides cloud-optimized datasets and scalable evaluation tools to compare data-driven and numerical models. It features deterministic and probabilistic metrics, tools for computing derived meteorological variables, and supports large-scale processing via Apache Beam on Google Cloud Dataflow or Apache Spark. Note: Users starting new evaluation projects are encouraged to migrate to WeatherBench-X.

Tokens
16.6K
Snippets
34
Records
54
Agent score
70%

What's inside WeatherBench 2

  1. Overview of WeatherBench 2

    main

    WeatherBench 2 is a framework designed for evaluating and comparing data-driven and traditional numerical weather forecasting models. It provides a standardized environment for benchmarking global weather models using publicly available, cloud-optimized datasets.

    The framework consists of:

    • Datasets: Cloud-optimized ground truth and baseline datasets.
    • Evaluation Code: Open-source code designed for scalability, including command-line scripts that utilize Xarray-Beam for large-scale processing.
    • Leaderboard: A website displaying up-to-date scores for state-of-the-art approaches.

    Note for new users: An updated version of the evaluation codebase, WeatherBench-X, has been released. Users are encouraged to switch to the WeatherBench-X repository for evaluation, though the existing data guide remains valid.

  2. Overview of WeatherBench 2 components

    main

    WeatherBench 2 is a framework designed to evaluate and compare data-driven and traditional numerical weather forecasting models. It consists of four primary components:

    • Datasets: Publicly available, cloud-optimized ground truth and baseline datasets.
    • Evaluation Code: Open-source code for running evaluations. It is designed for scalability and can be run locally or on GCP using Xarray-Beam and DataFlow.
    • Leaderboard: A website displaying up-to-date scores for state-of-the-art models.
    • Research Foundation: A formal paper describing the evaluation setup and rationale.

  3. Run WeatherBench 2 pipelines using Apache Beam

    main

    All WeatherBench 2 pipelines are built using Apache Beam. This allows you to run the routines on different execution engines:

    • Google Cloud Dataflow: The recommended way to run pipelines in the examples provided.
    • Apache Spark: An alternative runner available on most major commercial clouds or via self-hosting.

    For detailed instructions on running Beam pipelines in the cloud, refer to the WeatherBench 2 Beam in the Cloud documentation.

  4. Understand how time_slice affects data selection in different conventions

    main

    The behavior of the time_slice argument in a Selection instance depends on the chosen time convention:

    Init-time convention

    time_slice refers to the initialization time. If you set time_slice = slice('2020-01-01', '2020-12-31'), the evaluation will include forecasts initialized during 2020. Because of the lead time, these forecasts will remain valid into the next year (e.g., a 10-day lead forecast initialized on 2020-12-31 will be valid until 2021-01-10).

    Valid-time convention

    time_slice refers to the valid time. If you set time_slice = slice('2020-01-01', '2020-12-31'), the evaluation will only include forecasts that are valid within the year 2020. This may include forecasts that were actually initialized in late 2019.

  7. Install WeatherBench 2

    main

    To use WeatherBench 2, clone the repository and install it using pip.

    Depending on your use case, choose one of the following installation methods:

    1. Standard Installation: For general use of the evaluation code.
    2. Development Installation: Use the editable flag (-e) if you intend to modify the source code.
    3. GCP Installation: If you plan to run evaluation jobs on Google Cloud Platform (GCP), install the [gcp] extra to include necessary dependencies.
    # Clone the repository
git clone git@github.com:google-research/weatherbench2.git

# Standard installation
cd weatherbench2
pip install .

# Development installation
pip install -e .

# Installation with GCP support
pip install .[gcp]

  8. Compute zonal energy spectra with compute_zonal_energy_spectrum.py

    main

    Use compute_zonal_energy_spectrum.py to calculate the zonal energy spectrum for a given dataset. This is useful for analyzing the spectral properties of weather models or reanalysis data.

    Required/Common arguments:

    • --input_path: Path to the input Zarr dataset.
    • --output_path: Path where the resulting spectra Zarr will be saved.
    • --time_start / --time_stop: The year(s) to include in the spectrum calculation.
    • --base_variables: A comma-separated list of variables to include in the spectrum calculation.

    Note: Variable names may vary depending on the dataset (e.g., some datasets use level-specific names like geopotential_500).

    # Example: Computing spectra for ERA5 data
python compute_zonal_energy_spectrum.py -- \
 --input_path=gs://weatherbench2/datasets/era5/1959-2022-6h-1440x721.zarr  \
 --output_path=$OUTDIR/highest_res/spectra/era_2020.zarr \
 --time_start=2020 \
 --time_stop=2020 \
 --base_variables=geopotential,specific_humidity,temperature,u_component_of_wind,v_component_of_wind,wind_speed,10m_u_component_of_wind,10m_v_component_of_wind,10m_wind_speed,2m_temperature,mean_sea_level_pressure,total_precipitation_6hr,total_precipitation_24hr \

  9. Save forecasts in the standardized Zarr format

    main

    The WeatherBench 2 evaluation pipeline requires both forecasts and ground truth to be stored in Zarr format.

    When preparing your data:

    • You can save all variables or select specific levels and variables to save space.
    • Chunking Strategy: For optimal performance, it is recommended to use single chunks in space (latitude and longitude) and small chunks in time. This configuration enables efficient parallelization during evaluation.

    Refer to the Pangu forecasts documentation for a concrete implementation example.

  10. Run deterministic evaluation with `evaluate.py`

    main

    Use the evaluate.py script to perform deterministic evaluations of weather models against observations (e.g., ERA5). The script supports multiple resolutions (64x32, 240x121, 1440x721) and allows for comparing forecasts against observations, climatology, or persistence.

    Key parameters include:

    • --forecast_path: Path to the model forecast Zarr dataset.
    • --obs_path: Path to the observation Zarr dataset.
    • --climatology_path: Path to the climatology Zarr dataset.
    • --eval_configs: Specifies evaluation types, such as deterministic or deterministic_temporal.
    • --variables: A comma-separated list of weather variables to evaluate.
    • --time_start / --time_stop: The temporal range for the evaluation.
    • --output_dir: Directory where results will be stored.
    • --output_file_prefix: Prefix for the generated output files.
    python evaluate.py \
  --forecast_path=gs://weatherbench2/datasets/hres/2016-2022-0012-64x32_equiangular_conservative.zarr \
  --obs_path=gs://weatherbench2/datasets/era5/1959-2022-6h-64x32_equiangular_conservative.zarr \
  --climatology_path=gs://weatherbench2/datasets/era5-hourly-climatology/1990-2019_6h_64x32_equiangular_conservative.zarr \
  --output_dir=$OUTDIR/64x32/deterministic/ \
  --output_file_prefix=hres_vs_era_2020_ \
  --input_chunks=init_time=1 \
  --fanout=27 \
  --regions=all \
  --eval_configs=deterministic,deterministic_temporal \
  --evaluate_climatology=False \
  --evaluate_persistence=False \
  --time_start=2020-01-01 \
  --time_stop=2020-12-31 \
  --variables=geopotential,temperature,u_component_of_wind,v_component_of_wind,specific_humidity,2m_temperature,10m_u_component_of_wind,10m_v_component_of_wind,mean_sea_level_pressure,total_precipitation_6hr,total_precipitation_24hr,10m_wind_speed,wind_speed \
  --compute_seeps=True \
  --use_beam=True

  11. Migrate to WeatherBench-X for evaluation

    main
    If you are starting a new evaluation project, the developers recommend using WeatherBench-X instead of the original WeatherBench 2 codebase. WeatherBench-X provides an updated version of the evaluation logic. However, the data documentation and guides provided by WeatherBench 2 are still considered up-to-date and applicable.

  12. Regrid forecasts to supported resolutions

    main

    While the WeatherBench 2 evaluation code can handle any resolution, the provided ground truth data is only available for specific resolutions.

    Supported resolutions include:

    • 1.5 degrees (the official resolution used in the paper)
    • 0.25 degrees
    • 5.625 degrees

    Important: Absolute metric values computed at different resolutions are not necessarily comparable.

    Regridding Method: Use the provided regridding script to prepare your data. The framework follows WMO standards by using conservative regridding.