DOCSEARCHbyAlgolia
Submit Docs

xee

repository·main·Indexed 16 days ago

https://github.com/google/xee

An Xarray backend for Google Earth Engine that enables lazy, parallel access to ee.Image and ee.ImageCollection objects. Xee allows developers to treat Earth Engine collections as standard Xarray Datasets for large-scale geospatial analysis using the Python scientific stack, including Dask. It supports flexible output grid definitions, CF-friendly dimension order [time, y, x], and integration with Google Cloud Dataflow for exporting ImageCollections to Zarr format.

Tokens
20.4K
Snippets
59
Records
101
Agent score
62%

What's inside xee

  1. What is Xee and why use it?

    main

    Xee is a library designed to bridge the gap between the Xarray ecosystem and Google Earth Engine (EE). It allows users to access EE-curated datasets as Xarray objects, making Earth Engine data compatible with the wider scientific Python ecosystem.

    Key Benefits

    • Xarray Integration: Access Earth Engine data using standard Xarray idioms and backends.
    • Quota Management: Provides mechanisms to avoid Earth Engine quota limits when computing pixels.
    • Flexible Data Shaping: Unlike Zarr, where data must be shaped up-front, Xee leverages Earth Engine's pyramided datasets to allow users to request specific resolutions and projections on the fly during dataset opening.
    • Scalable Querying: Supports querying Earth Engine data at scale, including integration with Xarray-Beam and easy export to Zarr.

  2. What is Xee?

    main

    Xee is an Xarray backend for Google Earth Engine. It allows you to open ee.Image and ee.ImageCollection objects as lazy xarray.Datasets, enabling petabyte-scale Earth data analysis using the scientific Python stack (Xarray, Dask, etc.).

    Key Features:

    • Lazy, parallel pixel retrieval: Data is fetched through Earth Engine only when needed.
    • Flexible output grid definition: Supports both fixed resolution and fixed shape.
    • CF-friendly dimension order: Uses [time, y, x].
    • Ecosystem integration: Works seamlessly with Xarray and Dask.

  3. How the Xee call chain works

    main

    Xee acts as a bridge between Xarray and Google Earth Engine through a specific internal hierarchy:

    1. User API: You call xarray.open_dataset(..., engine='ee').
    2. Entrypoint: Xarray routes the call to xee.EarthEngineBackendEntrypoint.open_dataset.
    3. Backend Store: The entrypoint creates an internal xee.EarthEngineStore which handles the actual streaming of pixels and metadata from Earth Engine.

    Most users should interact only with the xr.open_dataset public API.

  4. Access the Core extension backend for advanced workflows

    main

    For advanced users or those debugging the Xarray-Earth Engine integration, Xee exposes the core backend interfaces. While most users should stick to the high-level xarray.open_dataset(..., engine='ee') interface, these components allow for deeper control over how Earth Engine data is mapped to Xarray arrays.

    Core components include:

    • EarthEngineBackendEntrypoint: The entry point for the Xarray backend.
    • EarthEngineStore: Manages the storage and retrieval of Earth Engine data.
    • EarthEngineBackendArray: The underlying array implementation used by the backend.

  5. Choose between `grid_shape` and `grid_scale`

    main

    When opening datasets with Xee, you must decide how to define the spatial grid:

    • Use grid_shape when you need a fixed number of pixels (e.g., for machine learning model inputs where a specific input dimension is required).
    • Use grid_scale when the physical resolution (e.g., meters per pixel) is the priority (e.g., when aligning data with 30m Landsat imagery).

  6. Consider CRS units and geographic distortion

    main

    All scale and translation values in crs_transform are expressed in the units of the specified crs.

    • Projected CRSs (e.g., EPSG:3857, UTM): Use linear units like meters or feet.
    • Geographic CRSs (e.g., EPSG:4326): Use angular units like degrees.

    Warning on Geographic Distortion: When using Geographic CRSs (like EPSG:4326), pixels are defined in degrees. Because the ground distance of a degree of longitude changes based on latitude, a grid in this CRS will have non-uniform ground pixel sizes.

    If your analysis requires uniform measurement of distance or area, you should reproject to a projected CRS (meters) suitable for your region of interest to avoid incorrect Euclidean calculations.

  7. Define the output pixel grid using Pixel Grid Parameters

    main

    To open Google Earth Engine (EE) data with Xee, you must specify an output pixel grid using three parameters:

    • crs: The Coordinate Reference System for the output grid (e.g., EPSG:4326, EPSG:32610).
    • crs_transform: An affine transform tuple (x_scale, x_skew, x_trans, y_skew, y_scale, y_trans) describing pixel size, rotation/skew, and origin translation in CRS units. This follows the Rasterio/affine.Affine standard.
    • shape_2d: The (width, height) of the output grid in pixels.

    Important Note on crs_transform ordering: Xee uses the order (a, b, c, d, e, f) where:

    • a: Scale X (pixel width)
    • b: Shear X (row rotation)
    • c: Translation X (x-origin)
    • d: Shear Y (column rotation)
    • e: Scale Y (pixel height, usually negative)
    • f: Translation Y (y-origin)

    This differs from the GDAL GeoTransform sequence (c, a, b, f, d, e). If mapping from GDAL, ensure translation indices 0 and 3 are mapped to indices 2 and 5 in Xee.

  8. Understand dimension ordering and collection types

    main

    Dimension Ordering

    Datasets returned by Xee follow CF conventions and are ordered as [time, y, x].

    Stored vs Computed Collections

    • Stored Collections: Unmodified ee.ImageCollection('ID'). It is recommended to use the high-volume endpoint for these to maximize throughput.
    • Computed Collections: Collections resulting from operations like .map(), .select(), filtering, or band math. The standard endpoint is often more efficient for these due to caching.

  9. Note on breaking changes in v0.1.0+

    main
    Xee v0.1.0 introduced a refactored API with breaking changes relative to the 0.0.x series. This documentation is written for the v0.1.0+ API. If you are upgrading from a 0.0.x version, you must refer to the migration-guide-v0.1.0.md to update your code.

  10. Optimize time slicing with fast_time_slicing

    main

    The fast_time_slicing parameter (default False) controls how time slices are resolved.

    • fast_time_slicing=False (Default): Xee slices directly from the in-memory Earth Engine ImageCollection object. This is the safest option for computed/modified collections (e.g., those using .map(), band math, or clipping).
    • fast_time_slicing=True: Xee slices by system:id first and then loads by those IDs. This is significantly faster for direct/stored collections but can bypass computed modifications, returning the original asset images instead of your transformed ones.

    Recommendation: Use False for correctness-sensitive workflows involving transformations. Enable True only for direct asset reads after validating outputs.

  11. Specify output geography using grid parameter helpers

    main

    In v0.1.0, xr.open_dataset requires explicit grid parameters: crs, crs_transform, and shape_2d. Instead of manually calculating these, use the xee.helpers module.

    Option 1: Match the source grid (Recommended)

    Use helpers.extract_grid_params(ee_obj) to automatically retrieve the native grid parameters from an Earth Engine Image or ImageCollection.

    Option 2: Fit geometry with a specific scale

    Use helpers.fit_geometry with the grid_scale parameter. Note that for north-up orientation, the y-scale should typically be negative.

    Option 3: Fit geometry with a specific pixel shape

    Use helpers.fit_geometry with the grid_shape parameter to define the output grid by its width and height in pixels.

    import ee
import xarray as xr
from xee import helpers
import shapely

# Option 1: Match Source Grid
ic = ee.ImageCollection('ECMWF/ERA5_LAND/MONTHLY_AGGR')
grid_params = helpers.extract_grid_params(ic)
ds = xr.open_dataset(ic, engine='ee', **grid_params)

# Option 2: Fit Geometry with Specific Scale
aoi = shapely.geometry.box(-180, -90, 180, 90)
grid_params = helpers.fit_geometry(
    geometry=aoi,
    grid_crs='EPSG:4326',
    grid_scale=(0.25, -0.25)
)
ds = xr.open_dataset('ECMWF/ERA5_LAND/MONTHLY_AGGR', engine='ee', **grid_params)

# Option 3: Fit Geometry with Specific Shape
aoi = shapely.geometry.box(113.33, -43.63, 153.56, -10.66)
grid_params = helpers.fit_geometry(
    geometry=aoi,
    grid_crs='EPSG:4326',
    grid_shape=(256, 256)
)
ds = xr.open_dataset('ECMWF/ERA5_LAND/MONTHLY_AGGR', engine='ee', **grid_params)

  12. Understand the new dimension ordering [time, y, x]

    main

    Xee v0.1.0 changed the default dimension order from [time, x, y] to [time, y, x] to align with CF conventions and most geospatial tools.

    Impact on Workflows

    • Plotting: You no longer need to call .transpose() before plotting with Xarray. Data plots correctly by default.
    • Library Integration: Most geospatial libraries expecting [time, y, x] can now consume the DataArray directly without manual transposition.
    • Explicit Access: If your code relies on dimension indices (e.g., ds.dims[1] being x), you must update it to reflect that index 1 is now y and index 2 is x.

    Best Practice: Use dimension-agnostic access to avoid version-specific issues:

    width = ds.sizes['x']
height = ds.sizes['y']
time_length = ds.sizes['time']