openEO Python Client
repository·master·Indexed 42 days ago
https://github.com/open-eo/openeo-python-clientA Python library for interacting with openEO backends to perform remote geospatial data processing and analysis. It provides a high-level interface via `openeo.connect` for concise workflows and a low-level REST interface via `openeo.rest` for granular control. Key features include support for OIDC authentication, data cube management, User Defined Functions (UDFs), and an artifacts API for staging files in processing environments.
What's inside openEO
- The openEO Federation extension is a set of additional specifications built on top of the standard openEO API. It provides extra metadata required for federated openEO processing, which occurs when multiple separately operated openEO services are bundled together behind a single API endpoint.
Understand the source of Spectral Index specifications
masterThe spectral index specifications used in this project are derived from the Awesome Spectral Indices project by David Montero Loaiza.
Specifically:
- The
awesome-spectral-indices/directory contains specifications based on a versioned release of the original project. - The
extra-indixes-dict.jsonfile contains additional spectral indices that are not yet included in the primary Awesome Spectral Indices dataset.
- The
How DataCube operations and Band Math work
masterThe
DataCubeobject allows you to perform complex operations using mathematical operators and method calls.- Band Selection: Use
.band("BAND_NAME")to select a specific band. - Band Math: You can use standard arithmetic operators (
+,-,*,/) on band objects. This is 'syntactic sugar' that builds an abstract openEO process graph (a JSON representation of the algorithm) rather than performing immediate computation. - Convenience Methods: The library provides helper methods like
.max_time()(which implements a temporalreducewith amaxaggregator) to simplify common patterns.
Actual data processing only occurs when you explicitly trigger execution (e.g., via
.download()).# Select and rescale bands blue = sentinel2_cube.band("B02") * 0.0001 red = sentinel2_cube.band("B04") * 0.0001 nir = sentinel2_cube.band("B08") * 0.0001 # Compute an index (EVI) evi_cube = 2.5 * (nir - red) / (nir + 6.0 * red - 7.5 * blue + 1.0) # Use a convenience method for temporal reduction evi_composite = evi_cube.max_time()- Band Selection: Use
Migrate to the new `openeo.UDF` API
masterSince version 0.13.0, the way UDFs are handled has been simplified.
Key Changes:
- Auto-detection:
openeo.UDFnow automatically detects theruntimefrom the source code or file extension. You no longer need to passruntime="Python"explicitly. - Simplified Arguments: The
dataargument is deprecated and unused; you should no longer provide it. - Uniformity: Use the
processargument for all operations (apply,reduce_dimension,apply_dimension) instead of specialized arguments likecodeorruntime.
Recommended Pattern:
# Create the UDF object udf = openeo.UDF(udf_code) # Use the 'process' argument in DataCube methods cube.apply(process=udf) cube.reduce_dimension(reducer=udf, dimension="t") cube.apply_dimension(process=udf, dimension="t")- Auto-detection:
How the artifacts API works
masterThe artifacts functionality automates the process of uploading files to provider-managed storage (such as S3-compatible object storage) so they can be used during job execution.
- Factory Pattern:
build_artifact_helperinspects yourconnectionobject to see if the openEO backend advertises a preferred configuration type and returns the appropriate helper. - Storage & Access:
upload_fileretrieves temporary credentials (e.g., S3 credentials) based on the backend's advertised config and your current session, then uploads the file and returns a URI that the backend can resolve.get_presigned_urlcreates a standalone URI signed with those temporary credentials. This is useful for tools or execution steps that cannot handle internal backend references.
- Factory Pattern:
Use Batch Jobs for large processing tasks
masterFor large data cubes or complex algorithms that might exceed computation time limits, use Batch Jobs instead of synchronous downloads.
Batch jobs allow for asynchronous execution: you can start a job, disconnect from the back-end, and reconnect later to check the status or download the results. The library provides helpers to monitor running jobs and view progress reports.
Perform Random Forest Classification and Regression in openEO
masteropenEO provides specific processes for Random Forest-based machine learning workflows for Earth Observation applications:
fit_class_random_forest: Trains a random forest classification model.fit_regr_random_forest: Trains a random forest regression model.predict_random_forest: Performs inference/prediction using a trained model.
The openEO Python Client facilitates setting up and executing these training and inference workflows as batch jobs.
Warning: This API and documentation are experimental and subject to change.
Implement UDF metadata transformations
masterIf your UDF changes the metadata of a cube (e.g., adjusting spatial extent), the openEO backend might not be able to infer this automatically. To allow the backend to validate process graphs or estimate data sizes, you should implement theapply_metadata()function within your UDF script. This is an experimental feature.How MultiBackendJobManager works
masterThe
MultiBackendJobManageris designed to automate the execution of large numbers of batch jobs across one or more openEO backends. It is particularly useful for processing large areas of interest by splitting them into smaller tiles and running a job for each tile.To use the manager, you need three components:
- A Manager: An instance of
MultiBackendJobManagerwith one or more registered backends. - A Job Database: A
pandas.DataFrame(wrapped in a persistent database) where each row represents a single job and contains the parameters needed for that job. - A
start_jobcallback: A user-defined function that takes a row from the database and a connection, and returns an openEOBatchJobobject.
The manager handles job creation, submission, status tracking, result downloading, error handling, and metadata persistence automatically.
import openeo from openeo.extra.job_management import MultiBackendJobManager manager = MultiBackendJobManager() manager.add_backend("cdse", connection=openeo.connect( "https://openeo.dataspace.copernicus.eu/" ).authenticate_oidc())- A Manager: An instance of
Choose between High-level and Low-level APIs
masterThe openEO Python client offers two distinct ways to interact with back-ends:
- High-level Interface: Uses the
openeomodule (e.g.,openeo.connect). It is designed to be concise and readable, abstracting away the underlying web service details. - Low-level Interface: Uses the
openeo.restmodules. This provides more granular control over the API calls, resource management, and specific RESTful interactions with the server.
- High-level Interface: Uses the
How UDFs work with apply and reduce meta-processes
masterUDFs are typically used as "callbacks" for meta-processes. The meta-process manages the data chunking and orchestration, while the UDF performs the specific logic on the data provided to it:
apply: Applies the process to each pixel separately. Dimensions and labels are fully preserved.apply_dimension: Applies the process along a specific dimension. The backend will not split data on this dimension (e.g., if applying alongtime, you receive a full timeseries). All dimensions are preserved except the target dimension, which may change its number of labels.reduce_dimension: Applies the process along a dimension to produce a single value, effectively removing that dimension from the output.apply_neighborhood: Applies the process to a neighborhood of pixels using a sliding window. This is the most versatile and common way to use UDFs. User controls the chunking explicitly here.
How processes with child callbacks work
masterSome openEO processes act as 'parent' processes that require a sub-process (a callback) to be applied to a subset or slice of the data. The parent process manages the execution and invocation of the callback.
Common parent processes include:
apply: Requires a transformation applied to each pixel separately (e.g., scaling a value).reduce_dimension: Requires an aggregation function to convert an array of values (along a dimension) into a single value (e.g., temporal mean).aggregate_spatial: Requires a function to aggregate values within specific geometries.
Supported
DataCubemethods that accept callbacks include:aggregate_spatialaggregate_temporalapplyapply_dimensionapply_neighborhoodreduce_dimension