xbatcher

Batch generation in Xbatcher is driven by a schema. Currently, the schema is a dictionary where keys are the names of the dimensions you wish to batch over, and values are the integer size of the batch for that dimension.

Example schema: {'time': 10} tells the generator to slice the time dimension into chunks of size 10.

Once your changes are ready, follow these steps to submit a Pull Request:

1. Stage and commit your changes: Ensure your commit message has a subject line under 72 characters, followed by a blank line and an optional body.

   git add path/to/file.py
   git commit -m "Your descriptive commit message"

2. Push to your fork: Push the branch to your remote repository (origin).

   git push origin shiny-new-feature

3. Open a Pull Request: Navigate to your branch on GitHub; a banner will appear allowing you to submit a pull request to the xarray-contrib/xbatcher repository.

Documentation is built from docstrings and files in the doc/ directory. To build it locally, create an environment using the documentation requirements:

1. Create and activate the docs environment:

conda env create --file ci/requirements/docs.yml
conda activate xbatcher-docs

2. Build the HTML documentation:

cd docs
make html

xbatcher uses pytest for testing. To run the full test suite, execute:

pytest xbatcher

The project uses pre-commit to manage code linting and style. After activating your Conda development environment, install the pre-commit hooks by running:

pre-commit install

Always work on a feature branch rather than main.

Create a new branch:

git checkout -b shiny-new-feature

Update your branch with the latest changes from upstream: To incorporate the latest code from the main project, fetch the upstream changes and merge them into your current branch. If you have uncommitted changes, use git stash before merging, then reapply them after.

git fetch upstream
git merge upstream/main

To use xbatcher with machine learning frameworks, you can install the optional dependencies using extra install tags via pip or by including the framework in your conda command.

PyTorch

pip:

python -m pip install xbatcher[torch]

conda:

conda install -c conda-forge xbatcher pytorch

TensorFlow

pip:

python -m pip install xbatcher[tensorflow]

conda:

conda install -c conda-forge xbatcher tensorflow

You can install xbatcher using pip, conda, or directly from the GitHub source.

Using pip

python -m pip install xbatcher

Using Conda

conda install -c conda-forge xbatcher

From source

python -m pip install git+https://github.com/xarray-contrib/xbatcher.git

Performance benchmarking is managed via asv. Benchmarks are located in the asv_bench/ directory.

1. Install asv:

pip install git+https://github.com/airspeed-velocity/asv

2. Run continuous benchmarks: To run benchmarks comparing main against your current branch:

# Using default conda backend
asv continuous -f 1.1 main <my-branch>

# Using virtualenv backend
asv continuous -f 1.1 -E virtualenv main <my-branch>

3. Run specific benchmark groups: Use a dot . as a separator to run a specific benchmark defined in a file (e.g., benchmarks.py):

asv continuous -f 1.1 main HEAD -b benchmarks.Generator.time_batch_preload

Options:

• -E virtualenv: Forces asv to use virtualenv instead of the default conda backend.

To contribute to xbatcher, you must first fork the project on GitHub. Once forked, clone your personal fork to your local machine and add the original repository as an upstream remote to keep your fork synchronized with the main project.

# Clone your fork
git clone git@github.com:yourusername/xbatcher.git
cd xbatcher

# Add the upstream remote
git remote add upstream git@github.com:xarray-contrib/xbatcher.git

To test code changes, build xbatcher from source using a Conda environment. This process uses the requirements defined in the repository's CI configuration.

1. Create and activate the build environment:

conda env create --file ci/requirements/environment.yml
conda activate xbatcher-tests

2. Verify the installation by importing xbatcher in a Python interpreter:

import xbatcher
print(xbatcher.__version__)

To manage your environments, use conda info --envs to view them or conda deactivate to return to your base environment.