openEO Python Client

repository·master·Indexed 42 days ago

https://github.com/open-eo/openeo-python-client

A Python library for interacting with openEO backends to perform remote geospatial data processing and analysis. It provides a high-level interface via `openeo.connect` for concise workflows and a low-level REST interface via `openeo.rest` for granular control. Key features include support for OIDC authentication, data cube management, User Defined Functions (UDFs), and an artifacts API for staging files in processing environments.

Tokens
55.2K
Snippets
180
Records
268
Agent score
52%

What's inside openEO

  1. Understand the source of Spectral Index specifications

    master

    The spectral index specifications used in this project are derived from the Awesome Spectral Indices project by David Montero Loaiza.

    Specifically:

    • The awesome-spectral-indices/ directory contains specifications based on a versioned release of the original project.
    • The extra-indixes-dict.json file contains additional spectral indices that are not yet included in the primary Awesome Spectral Indices dataset.
  2. How DataCube operations and Band Math work

    master

    The DataCube object allows you to perform complex operations using mathematical operators and method calls.

    • Band Selection: Use .band("BAND_NAME") to select a specific band.
    • Band Math: You can use standard arithmetic operators (+, -, *, /) on band objects. This is 'syntactic sugar' that builds an abstract openEO process graph (a JSON representation of the algorithm) rather than performing immediate computation.
    • Convenience Methods: The library provides helper methods like .max_time() (which implements a temporal reduce with a max aggregator) to simplify common patterns.

    Actual data processing only occurs when you explicitly trigger execution (e.g., via .download()).

    # Select and rescale bands
    blue = sentinel2_cube.band("B02") * 0.0001
    red = sentinel2_cube.band("B04") * 0.0001
    nir = sentinel2_cube.band("B08") * 0.0001
    
    # Compute an index (EVI)
    evi_cube = 2.5 * (nir - red) / (nir + 6.0 * red - 7.5 * blue + 1.0)
    
    # Use a convenience method for temporal reduction
    evi_composite = evi_cube.max_time()
  3. Migrate to the new `openeo.UDF` API

    master

    Since version 0.13.0, the way UDFs are handled has been simplified.

    Key Changes:

    • Auto-detection: openeo.UDF now automatically detects the runtime from the source code or file extension. You no longer need to pass runtime="Python" explicitly.
    • Simplified Arguments: The data argument is deprecated and unused; you should no longer provide it.
    • Uniformity: Use the process argument for all operations (apply, reduce_dimension, apply_dimension) instead of specialized arguments like code or runtime.

    Recommended Pattern:

    # Create the UDF object
    udf = openeo.UDF(udf_code)
    
    # Use the 'process' argument in DataCube methods
    cube.apply(process=udf)
    cube.reduce_dimension(reducer=udf, dimension="t")
    cube.apply_dimension(process=udf, dimension="t")
  4. How the artifacts API works

    master

    The artifacts functionality automates the process of uploading files to provider-managed storage (such as S3-compatible object storage) so they can be used during job execution.

    1. Factory Pattern: build_artifact_helper inspects your connection object to see if the openEO backend advertises a preferred configuration type and returns the appropriate helper.
    2. Storage & Access:
      • upload_file retrieves temporary credentials (e.g., S3 credentials) based on the backend's advertised config and your current session, then uploads the file and returns a URI that the backend can resolve.
      • get_presigned_url creates a standalone URI signed with those temporary credentials. This is useful for tools or execution steps that cannot handle internal backend references.
  5. Use Batch Jobs for large processing tasks

    master

    For large data cubes or complex algorithms that might exceed computation time limits, use Batch Jobs instead of synchronous downloads.

    Batch jobs allow for asynchronous execution: you can start a job, disconnect from the back-end, and reconnect later to check the status or download the results. The library provides helpers to monitor running jobs and view progress reports.

  6. Perform Random Forest Classification and Regression in openEO

    master

    openEO provides specific processes for Random Forest-based machine learning workflows for Earth Observation applications:

    • fit_class_random_forest: Trains a random forest classification model.
    • fit_regr_random_forest: Trains a random forest regression model.
    • predict_random_forest: Performs inference/prediction using a trained model.

    The openEO Python Client facilitates setting up and executing these training and inference workflows as batch jobs.

    Warning: This API and documentation are experimental and subject to change.

  7. Implement UDF metadata transformations

    master
    If your UDF changes the metadata of a cube (e.g., adjusting spatial extent), the openEO backend might not be able to infer this automatically. To allow the backend to validate process graphs or estimate data sizes, you should implement the apply_metadata() function within your UDF script. This is an experimental feature.
  8. How MultiBackendJobManager works

    master

    The MultiBackendJobManager is designed to automate the execution of large numbers of batch jobs across one or more openEO backends. It is particularly useful for processing large areas of interest by splitting them into smaller tiles and running a job for each tile.

    To use the manager, you need three components:

    1. A Manager: An instance of MultiBackendJobManager with one or more registered backends.
    2. A Job Database: A pandas.DataFrame (wrapped in a persistent database) where each row represents a single job and contains the parameters needed for that job.
    3. A start_job callback: A user-defined function that takes a row from the database and a connection, and returns an openEO BatchJob object.

    The manager handles job creation, submission, status tracking, result downloading, error handling, and metadata persistence automatically.

    import openeo
    from openeo.extra.job_management import MultiBackendJobManager
    
    manager = MultiBackendJobManager()
    manager.add_backend("cdse", connection=openeo.connect(
        "https://openeo.dataspace.copernicus.eu/"
    ).authenticate_oidc())
  9. Choose between High-level and Low-level APIs

    master

    The openEO Python client offers two distinct ways to interact with back-ends:

    1. High-level Interface: Uses the openeo module (e.g., openeo.connect). It is designed to be concise and readable, abstracting away the underlying web service details.
    2. Low-level Interface: Uses the openeo.rest modules. This provides more granular control over the API calls, resource management, and specific RESTful interactions with the server.
  10. How UDFs work with apply and reduce meta-processes

    master

    UDFs are typically used as "callbacks" for meta-processes. The meta-process manages the data chunking and orchestration, while the UDF performs the specific logic on the data provided to it:

    • apply: Applies the process to each pixel separately. Dimensions and labels are fully preserved.
    • apply_dimension: Applies the process along a specific dimension. The backend will not split data on this dimension (e.g., if applying along time, you receive a full timeseries). All dimensions are preserved except the target dimension, which may change its number of labels.
    • reduce_dimension: Applies the process along a dimension to produce a single value, effectively removing that dimension from the output.
    • apply_neighborhood: Applies the process to a neighborhood of pixels using a sliding window. This is the most versatile and common way to use UDFs. User controls the chunking explicitly here.
  11. How processes with child callbacks work

    master

    Some openEO processes act as 'parent' processes that require a sub-process (a callback) to be applied to a subset or slice of the data. The parent process manages the execution and invocation of the callback.

    Common parent processes include:

    • apply: Requires a transformation applied to each pixel separately (e.g., scaling a value).
    • reduce_dimension: Requires an aggregation function to convert an array of values (along a dimension) into a single value (e.g., temporal mean).
    • aggregate_spatial: Requires a function to aggregate values within specific geometries.

    Supported DataCube methods that accept callbacks include:

    • aggregate_spatial
    • aggregate_temporal
    • apply
    • apply_dimension
    • apply_neighborhood
    • reduce_dimension