climpred

climpred is a package designed for analyzing output from initialized dynamical forecast models, covering scales from short-term weather forecasts to decadal climate forecasts.

Core Capabilities:

• Forecast Assessment: Provides tools to assess forecasts against validation products like observations, reanalysis products, control simulations, or baseline forecasts.
• Verification Metrics: Includes deterministic and probabilistic verification metrics (e.g., mean absolute error, rank histograms) and advanced contingency table-derived metrics.
• Data Handling: Uses xarray to manage core prediction ensemble dimensions (ensemble member, initialization date, and lead time) and dask for out-of-memory computations on large datasets.

User Responsibility: Users are expected to perform their own domain-specific post-processing of model output before using climpred for analysis.

Probabilistic metrics in climpred account for the spread of ensemble simulations and assign probability values between 0 and 1 to forecasts. To use these metrics, pass the corresponding keyword to the metric= argument in compute functions.

Available probabilistic metrics and their keywords:

• Continuous Ranked Probability Score (CRPS): crps
• Continuous Ranked Probability Skill Score (CRPSS): crpss
• Continuous Ranked Probability Skill Score Ensemble Spread: crpss_es
• Brier Score: brier_score
• Threshold Brier Score: threshold_brier_score
• Ranked Probability Score: rps
• Reliability: reliability
• Discrimination: discrimination
• Rank Histogram: rank_histogram
• Logarithmic Ensemble Spread Score: less

# Example of how keywords are used (conceptually)
# print(f"Keywords: {metric_aliases['crps']}")
# Use 'crps' in your compute function call

climpred uses an object-oriented approach to verify different types of prediction systems. The primary abstraction is the PredictionEnsemble parent class, which is implemented through two specialized subclasses:

1. HindcastEnsemble: Used for traditional hindcasts. It is instantiated with an initialized hindcast ensemble dataset and requires an observational dataset for verification.
2. PerfectModelEnsemble: Used for the "perfect-model" framework (investigating theoretical predictability limits). It is instantiated with an initialized perfect-model ensemble dataset and accepts a control dataset for evaluation.

Both types can optionally track an uninitialized dataset, which represents a historical simulation evolving via internal climate variability or used to isolate external forcing.

# Example of the hierarchy (conceptual)
# PredictionEnsemble
# ├── HindcastEnsemble (requires observations)
# └── PerfectModelEnsemble (requires control)

In climpred, simulation designs are categorized into two main ensemble types, which dictate how members are initialized and used for verification:

1. Hindcast Ensemble (HindcastEnsemble): Ensemble members are initialized from a simulation (typically a reconstruction from reanalysis) or an analysis at specific initialization dates and integrated for a set number of lead years. Use HindcastEnsemble.add_observations to provide reconstruction/assimilation data.
2. Perfect Model Experiment (PerfectModelEnsemble): Ensemble members are initialized from a control simulation (added via PerfectModelEnsemble.add_control) at randomly chosen initialization dates and integrated for lead years. This setup is used to estimate (Potential) Predictability by comparing the ensemble against the control simulation.

Additionally, you can work with Uninitialized Ensembles (HindcastEnsemble.add_uninitialized), which are generated by perturbing initial conditions only once in a historical run. These serve as a baseline to distinguish between internal climate variability and the forced response.

To use the PredictionEnsemble class, your xarray.Dataset must follow a specific naming convention for its dimensions. At a minimum, it requires the init and lead dimensions. If your dataset uses CF convention standard_name attributes that match these concepts, climpred will automatically rename them to the required internal names.

Required Dimensions

• init: Represents the initialization time steps (the start date of the experiment).
  • Type: pandas.DatetimeIndex or xarray.CFTimeIndex.
  • Note: If provided as an int, it is assumed to be annual data starting Jan 1st (this triggers a UserWarning).
  • CF Convention: forecast_reference_time.
• lead: Represents the lead time of the forecasts from initialization.
  • Type: int, float, or pandas.Timedelta (up to weeks).
  • Attribute Requirement: You must specify the units in the lead.attrs["units"] attribute.
  • Valid Units: "years", "seasons", "months", "weeks", "pentads", "days", "hours", "minutes", or "seconds".
  • CF Convention: forecast_period.
  • Note: If lead is a pandas.Timedelta up to weeks, it is converted to an int and the units are set automatically. For larger timedeltas (months, seasons, years), conversion is not possible.
• member (Optional/Conditional): Holds ensemble members. Required for probabilistic metrics.
  • Type: int or str.
  • CF Convention: realization.

Additional dimensions like lat, lon, or depth will be broadcasted automatically.

To assess how well a forecasting system performs, climpred uses several metrics and concepts:

• Skill (HindcastEnsemble.verify): The current ability of the ensemble to predict the real world. It is calculated by comparing the prediction ensemble against observations, reanalysis, or analysis.
• Accuracy: The average correspondence between forecast pairs and observations. Common metrics include:
  • Mean Absolute Error (climpred.metrics._mae)
  • Mean Square Error (climpred.metrics._mse)
• Association: The strength of the relationship between forecasts and observations, typically measured via:
  • Anomaly Correlation Coefficient (ACC) using Pearson product-moment correlation (climpred.metrics._pearson_r)
  • Spearman's Rank correlation (climpred.metrics._spearman_r)
• Skill Score: A normalized measure of improvement over a reference forecast (like persistence). It is defined as: $$S = \frac{A_{f} - A_{r}}{A_{p} - A_{r}}$$ where $A_{f}$ is forecast accuracy, $A_{p}$ is perfect forecast accuracy, and $A_{r}$ is reference forecast accuracy.