WeatherBench-X

WeatherBench-X is a modular evaluation framework designed for the flexible evaluation of weather forecasts and ground truth data. It supports various data types, including sparse datasets from weather stations or satellites. It serves as the successor to the WeatherBench 2 evaluation code and powers the WeatherBench 2 benchmark.

Key technical characteristics:

• Modularity: Uses interoperable classes to define Data loaders, Interpolations, Metrics, and Aggregation.
• Xarray-based: All internal logic is built on xarray.DataArray objects.
• Scalability: Designed to handle very large datasets by splitting operations into small chunks, enabling scalable evaluation via Apache Beam.

WeatherbenchX provides tools for performing statistical inference on evaluation metrics computed from finite samples of weather predictions. Instead of relying solely on point estimates, these tools allow you to compute:

• Confidence intervals for a metric.
• Estimated standard errors (the standard deviation of the estimate under resampling).
• Significance tests (including p-values) to compare two models (testing the null hypothesis that the difference between their true underlying metrics is zero).
• Confidence intervals and standard errors for the difference between two models.

Note that these methods rely on statistical assumptions that may not hold for all metrics, data sources, or sample sizes. Use them with care and consider the specific assumptions of the chosen method.

Standard statistical methods often assume independent and identically distributed (IID) observations, which is frequently violated in weather forecasting due to temporal dependence between nearby forecast initialization times.

WeatherbenchX addresses this in two ways:

1. Temporal Dependence: Implements specific statistical methods (such as AR(2) correction, HAC estimators, and various bootstrap methods) designed to account for autocorrelation in time series.
2. Spatial Correlation: To avoid the complexity of spatial dependence, weatherbenchX typically treats spatial locations (e.g., grid points) as fixed and averages over them before performing statistical inference.

When choosing a method, consider the trade-offs between computational complexity and the ability to handle the specific dependence structure of your data.

After combining results, you can visualize the data using the following methods:

1. Scorecards: Use the WB_X_Website_Scorecard.ipynb notebook to plot scorecards.
2. Interactive Graphics: The code for the interactive Deterministic and Probabilistic tabs is located in the apps directory. Refer to the README within that directory for specific instructions on running the apps.

To begin using the framework, follow the official quickstart notebook which provides a guided introduction to the evaluation workflow.

Access the guide here: WeatherBench-X Quickstart Notebook

Install the WeatherBench-X package directly from the GitHub repository using pip.

!pip install git+https://github.com/google-research/weatherbenchX.git

To prepare evaluation results for downstream tasks like generating scorecards or interactive graphics, use combine_results.py. This script merges results into a single file.

Note that deterministic and probabilistic results must be processed separately using the --mode flag.

# Process deterministic results
python combine_results.py \
    --input_dir=gs://weatherbench2/benchmark_results \
    --output_dir=./ \
    --mode=deterministic

# Process probabilistic results
python combine_results.py \
    --input_dir=gs://weatherbench2/benchmark_results \
    --output_dir=./ \
    --mode=probabilistic

For testing or processing small datasets, you can run the evaluation pipeline on your local machine. Use the DirectRunner as the --runner argument. You can control the level of parallelism by setting the --direct_num_workers flag (passed after the -- separator).

python run_example_evaluation.py \
  --prediction_path=gs://weatherbench2/datasets/hres/2016-2022-0012-64x32_equiangular_conservative.zarr \
  --target_path=gs://weatherbench2/datasets/era5/1959-2022-6h-64x32_equiangular_conservative.zarr \
  --time_start=2020-01-01 \
  --time_stop=2020-01-02 \
  --output_path=./results.nc \
  --runner=DirectRunner \
  -- \
  --direct_num_workers 2

WeatherBench-X separates the per-element calculation from the final metric aggregation to allow for efficient processing of large datasets.

• Statistics (e.g., SquaredError) perform the heavy lifting by calculating values for every point in the prediction/target arrays. These are often PerVariableStatistic objects.
• Metrics (e.g., RMSE) act as a container. They specify a mapping of statistic names to Statistic objects via the statistics property.
• The metric then uses a method (like _values_from_mean_statistics_per_variable) to transform the averaged statistics into the final reported metric value. This allows you to perform operations like sqrt after the mean has been computed, which is mathematically different from taking the square root of every individual error before averaging.