DOCSEARCHbyAlgolia
Submit Docs

Herbie

repository·main·Indexed 21 days ago

https://github.com/blaylockbk/herbie

A Python package and CLI designed to simplify downloading and processing Numerical Weather Prediction (NWP) GRIB2 model data. It provides a unified interface to access data from sources such as NOAA, ECMWF, and AWS, allowing users to load meteorological data into xarray for analysis. Herbie supports over 15 weather models, including HRRR, RAP, GFS, and GEPS, and includes a painting module for meteorological visualization.

Tokens
29K
Snippets
102
Records
132
Agent score
72%

What's inside herbie-data

  1. How GRIB2 subsetting works in Herbie

    main

    Herbie performs subsetting by GRIB message (field) rather than by geographical region. GRIB2 files consist of multiple "messages" or "layers" stacked together.

    To avoid downloading massive files (e.g., a 700 MB HRRR file when you only need a 1 MB temperature field), Herbie uses HTTP Byte-Range requests via the cURL command.

    Key constraints and requirements:

    • Message-level only: You can subset by specific variables/fields, but you cannot perform regional (geographical) subsetting using this method. For regional subsetting, you must download the full file and use a tool like wgrib2's small_grib locally, or use the Zarr format.
    • Index Requirement: For Herbie to automate this, a remote index file (typically ending in .idx) must exist on the server. This index file maps variables to their specific byte ranges (start and end offsets).
    • Mechanism: Herbie identifies the requested variable's byte range from the index, executes a cURL command to fetch only those bytes, and can append multiple ranges to reconstruct a subsetted file.

  2. Understand Herbie data sources and search priority

    main

    Herbie discovers and downloads NWP (Numerical Weather Prediction) model output by searching across multiple remote servers. Because different sources have different availability (e.g., real-time vs. archived data), Herbie uses a default search priority to determine the order in which it looks for requested files.

    By default, Herbie searches sources in this order: aws → nomads → google → azure → pando → pando2.

    When requesting data, Herbie will automatically attempt to find the file in the highest priority source available. This is particularly important for managing the lifecycle of model data, such as when data on NOMADS is only available for a limited window (e.g., 2-14 days).

  3. Use Xarray accessors for model data analysis

    main

    Herbie provides specialized Xarray accessors that extend standard xarray.Dataset and xarray.DataArray objects with meteorological and geospatial capabilities. These accessors allow you to perform complex operations—such as selecting specific geographic domains, handling coordinate reference systems (CRS), and performing wind-related calculations—directly on your loaded model data objects.

    Available accessor tutorials include:

    • Pick Points: Selecting data at specific geographic points.
    • Cartopy CRS: Managing and transforming coordinate reference systems.
    • With Wind: Performing calculations involving wind vectors.
    • Domain Polygon: Subsetting data using geographic polygons.
    • Quick Maps: Rapidly generating geospatial visualizations.

  5. How `pick_points` works with curvilinear grids

    main

    Herbie's pick_points uses scikit-learn's BallTree algorithm with the haversine formula for accurate nearest-neighbor queries on Earth's spherical surface.

    This approach is specifically designed to handle:

    • Curvilinear/Projected Grids: Models like HRRR (Lambert Conformal) or NAM (WRF) that do not use simple lat/lon grids.
    • Longitude Convention Differences: Automatically handles differences between [0, 360) and [-180, 180) conventions.
    • Performance: Uses spatial indexing to allow fast queries even for thousands of points, and caches the BallTree (as a .pkl file) to speed up repeated queries on the same grid.

  7. How to extend Herbie with custom model templates

    main

    A model template in Herbie is a Python class that defines where Herbie looks for weather model datasets. You can extend Herbie to download additional types of model data if the following requirements are met:

    1. The NWP model data must exist on an HTTP server.
    2. File names must be predictable (consistent naming with date, model name, forecast lead time, product, etc.).
    3. Subsetting of a GRIB2 file requires an ASCII index or inventory file (preferably the wgrib2 style index file).

    Extension Methods

    • Public Template: Add a template directly to the Herbie source code by creating a class in herbie/models and importing it in herbie/models/__init__.py. This is intended for contributing new models to the main project via Pull Request.
    • Plugin Templates: Create a standalone Python package. When installed, Herbie automatically loads the templates provided by the plugin. This is ideal for accessing local GRIB2 files (e.g., WRF/MPAS output), accessing data on a private network, or overriding existing model behavior without modifying the Herbie core.

  8. Subset GRIB2 messages using the `search` method

    main

    Herbie allows for partial downloads of GRIB2 files by subsetting specific "layers" (GRIB messages) using byte-range requests. This is achieved via the search method, which uses regular expressions to match patterns in the file's index.

    Subsetting works by searching the H.inventory().search_this column of the inventory DataFrame. Herbie supports two types of index formats depending on the model provider:

    1. wgrib2-style: Common for NCEP models.
    2. ecCodes/grib_ls-style: Common for ECMWF models.

    Best Practices:

    • Use raw strings (e.g., r":TMP:2 m") for your regex patterns to prevent Python from interpreting backslashes as escape characters.
    • For complex selection logic that doesn't fit a single regex, you can use a "brute force" approach by joining multiple patterns with the regex OR operator (|).
    import herbie
# Example of brute force regex for specific hourly precipitation intervals
match_these = [f":APCP:surface:{i}-{i+1} h*" for i in range(47)]
search_pattern = f"({'|'.join(match_these)})"
# Use this pattern in your Herbie search call
# H.download(search=search_pattern)

  9. Use rclone CLI to access NOAA S3 buckets

    main

    You can access NOAA's public buckets on Amazon Web Services S3 using the syntax: rclone <command> <options> <remoteName>:<bucket>

    Common NOAA public buckets:

    DataBucket NameDocumentation
    HRRRnoaa-hrrr-bdp-pdslink
    GFSnoaa-gfs-bdp-pdslink
    GEFSnoaa-gefs-pdslink
    GOES16noaa-goes16link
    GOES17noaa-goes17link
    GOES18noaa-goes18link
    NEXRADnoaa-nexrad-level2link

    Note: bdp-pds stands for Big Data Program Public Data Set.

  10. Install Herbie via pip

    main

    You can install herbie-data using pip. To access full functionality, including xarray accessors for plotting and data manipulation, you must install the [extras] dependency group.

    # Install last release
pip install herbie-data

# Install with xarray accessors (recommended for full functionality)
pip install 'herbie-data[extras]'

# Install current main branch from GitHub
pip install git+https://github.com/blaylockbk/Herbie.git

  12. List and copy files with rclone

    main

    Use the following commands to navigate and download data from your configured remotes:

    List bucket directories: rclone lsd <remoteName>:<bucket>/

    List directories for specific folders: rclone lsd <remoteName>:<bucket>/<folder>

    List files in a bucket: rclone ls <remoteName>:<bucket>/<folder>

    Copy a file to your local machine: rclone copy <remoteName>:<bucket>/<path/to/file> <local_destination>

    # List directories
rclone lsd publicAWS:noaa-goes16/

# List files
rclone ls publicAWS:noaa-hrrr-bdp-pds/hrrr.20210101/conus

# Copy file to current directory
rclone copy publicAWS:noaa-goes16/ABI-L2-MCMIPC/2018/283/00/OR_ABI-L2-MCMIPC-M3_G16_s20182830057203_e20182830059576_c20182830100076.nc ./