DOCSEARCHbyAlgolia
Submit Docs

xskillscore

repository·main·Indexed 42 days ago

https://github.com/xarray-contrib/xskillscore

A Python package for verifying deterministic and probabilistic forecasts, built for the xarray ecosystem. It provides a Contingency object for calculating binary and multi-category metrics (such as Brier score, Heidke Skill Score, and Equitable Threat Score), comparative tests like the sign test and halfwidth CI test, and resampling utilities for bootstrapping via xskillscore.core.resampling.

Tokens
13.2K
Snippets
64
Records
75
Agent score
53%

What's inside xskillscore

  2. Use Contingency-based Metrics with the Contingency object

    main

    Contingency-based metrics in xskillscore require the construction of a Contingency object. This object builds a contingency table from forecasts and observations, and you can then call specific methods to retrieve metrics based on that table.

    Workflow

    1. Construct a Contingency object.
    2. Access the underlying table via Contingency.table.
    3. Call metric methods on the Contingency instance.

    Dichotomous-Only (yes/no) Metrics

    Available for binary outcomes:

    • Contingency.bias_score
    • Contingency.correct_negatives
    • Contingency.equit_threat_score
    • Contingency.false_alarm_rate
    • Contingency.false_alarm_ratio
    • Contingency.false_alarms
    • Contingency.hit_rate
    • Contingency.hits
    • Contingency.misses
    • Contingency.odds_ratio
    • Contingency.odds_ratio_skill_score
    • Contingency.success_ratio
    • Contingency.threat_score

    Multi-Category Metrics

    Available for multi-class outcomes:

    • Contingency.accuracy
    • Contingency.gerrity_score
    • Contingency.heidke_score
    • Contingency.peirce_score
    • roc

  3. Run tests in xskillscore

    main

    You can run various types of tests to ensure your changes are correct:

    • Standard tests: Run all tests using pytest.
    • Doctests: Test the code examples within docstrings.
    • Notebook tests: Test Jupyter notebooks by stripping output and building the HTML documentation.
    # Run all tests
pytest xskillscore

# Run doctests in docstrings (ignoring the tests directory)
pytest --doctest-modules xskillscore --ignore xskillscore/tests

# Test notebooks
cd docs
nbstripout source/*.ipynb
make -j4 html
    pytest xskillscore

pytest --doctest-modules xskillscore --ignore xskillscore/tests

cd docs
nbstripout source/*.ipynb
make -j4 html

  6. Build and lint documentation

    main

    Documentation is written in reStructuredText. To build it locally or lint files, use the following commands:

    Building HTML documentation:

    1. Update the documentation environment.
    2. Activate the environment.
    3. Run make html inside the docs directory.

    Linting reStructuredText: Run doc8 from the project root to lint .rst files.

    Adding new API functions to documentation: If you add new functions, run sphinx-autogen -o api api.rst from the docs/source directory and update api.rst.

    # Build documentation
conda env update -f ci/doc.yml
conda activate xskillscore-docs
cd docs
make html

# Linting
cd ..
doc8 *.rst

# Autogenerate API docs
sphinx-autogen -o api api.rst
    conda env update -f ci/doc.yml
conda activate xskillscore-docs
cd docs
make html

cd ..
doc8 *.rst

sphinx-autogen -o api api.rst

  7. Run performance benchmarks with ASV

    main

    xskillscore uses asv (Airspeed Velocity) to monitor performance regressions. Benchmarks are located in the asv_bench/ directory.

    To run a continuous benchmark comparing your branch against the upstream main branch:

    cd asv_bench/
asv continuous -f 1.1 upstream/main HEAD

    Filtering Benchmarks: Use the -b flag with a regular expression to run specific benchmarks or groups.

    • Specific file: asv continuous ... -b ^deterministic (runs benchmarks in deterministic.py).
    • Specific class and method: asv continuous ... -b deterministic.Compute_small.time_xskillscore_metric_small.
    asv continuous -f 1.1 upstream/main HEAD -b ^deterministic

asv continuous -f 1.1 upstream/main HEAD -b deterministic.Compute_small.time_xskillscore_metric_small

  8. Set up a development environment for xskillscore

    main

    To contribute to xskillscore, follow these steps to set up your local environment using mamba and conda:

    1. Fork and Clone: Fork the repository on GitHub, clone your fork, and add the upstream remote.
    2. Create Environment: Use the provided ci/dev.yml to update your conda environment.
    3. Install Package: Perform an editable install of the package.
    4. Run Pre-commit: Ensure code style consistency using pre-commit.
    # Clone and setup upstream
git clone git@github.com:YOUR_GITHUB_USERNAME/xskillscore.git
cd xskillscore
git remote add upstream git@github.com:xarray-contrib/xskillscore.git

# Create a feature branch
git checkout -b your-bugfix-feature-branch-name main

# Install dependencies
mamba env update -f ci/dev.yml
conda activate xskillscore-dev

# Editable install
pip install --no-deps -e .

# Run linting/formatting hooks
pre-commit run --all-files
    git clone git@github.com:YOUR_GITHUB_USERNAME/xskillscore.git
cd xskillscore
git remote add upstream git@github.com:xarray-contrib/xskillscore.git

# now, to fix a bug or add feature create your own branch off "main":

git checkout -b your-bugfix-feature-branch-name main

mamba env update -f ci/dev.yml
conda activate xskillscore-dev

pip install --no-deps -e .

pre-commit run --all-files

  10. Evaluate predictions over many columns efficiently

    main

    xskillscore is built on top of xarray.apply_ufunc, which provides vectorization. This makes xskillscore significantly faster than pandas.groupby.apply when dealing with large datasets or when evaluating predictions across many fields/dimensions.

    Performance Tip: Instead of using df.groupby([...]).apply(...), convert your DataFrame to an xarray.Dataset using .set_index([...]).to_xarray() and then use the xskillscore accessor. This approach leverages vectorized operations to speed up the calculation across the specified dimensions.

  11. Use xskillscore with tabular data (pandas DataFrames)

    main

    While xskillscore is designed for multidimensional arrays, it can be used effectively on tabular data (like pandas.DataFrame) by converting the DataFrame to an xarray.Dataset. This is particularly useful when you need to evaluate predictions across different fields or dimensions.

    To use xskillscore with tabular data:

    1. Prepare your pandas.DataFrame with target (y) and prediction (yhat) columns.
    2. Set the dimensions you want to group by as the index (e.g., using .set_index(['dim1', 'dim2'])).
    3. Convert to an xarray object using .to_xarray().
    4. Use the .xs accessor to call metrics, specifying the dimension to reduce (the original index) and using skipna=True to handle the nan values created by the conversion process.
    import pandas as pd
import xarray as xr
import xskillscore as xs

# 1. Prepare DataFrame
df = pd.DataFrame({
    'group': ['A', 'A', 'B', 'B'],
    'y': [10, 12, 20, 22],
    'yhat': [11, 11, 19, 23]
})

# 2. Set index and convert to xarray
# This creates dimensions for 'group' and an index dimension
ds = df.set_index(['group']).to_xarray()

# 3. Apply metric via .xs accessor
# We reduce the 'index' dimension to get a result per 'group'
out = ds.xs.mse('y', 'yhat', dim='index', skipna=True)

# 4. Convert back to pandas if needed
print(out.to_series())