sentle

repository·main·Indexed 41 days ago

A tool for downloading and processing large-scale Sentinel-1 and Sentinel-2 data cubes. It handles datasets larger than available memory by using parallel workers that write directly to Zarr stores. Key features include the `sentle.process` function for automated downloading, reprojection, and aggregation, as well as support for cloud/snow masking, NBAR correction, and temporal compositing.

Tokens: 2.5K
Snippets: 5
Records: 10
Agent score: 50%

What's inside sentle

Configure Sentinel-1 and Sentinel-2 data selection
main
You can control which satellite data is included in your Zarr cube:
- Sentinel-1: Controlled by S1_assets. Supported values are "vh_asc", "vh_desc", "vv_asc", and "vv_desc". Passing an empty list [] disables Sentinel-1.
- Sentinel-2: Controlled by S2_bands. By default, all 12 raw bands are downloaded. You can pass a subset (e.g., ["B04", "B03", "B02"]) to download only specific bands. Passing None or [] disables Sentinel-2.
Note: If you disable Sentinel-2 (S2_bands=None), you must provide S1_assets to create a Sentinel-1-only cube.
https://github.com/cmosig/sentle/blob/main/README.md
Configure temporal aggregation (compositing)
main
You can aggregate multiple acquisitions into a single time window by setting time_composite_freq.
- time_composite_freq: A string specifying the rounding interval (e.g., for daily or weekly aggregation).
- time_composite_method: Determines how pixels are aggregated within that window. Supported values are "mean" (default), "median", "min", or "max".
Warning: If you set a frequency but do not enable cloud or snow masking, you may get useless results for Sentinel-2 data due to unmasked clouds/snow being aggregated into the composite.
https://github.com/cmosig/sentle/blob/main/README.md
Configure Sentinel-2 masking and cloud classification
main
To use masking features, you must enable both the generation of the mask and the application of the mask:
- Snow Masking: Set S2_mask_snow=True to generate the mask, and S2_apply_snow_mask=True to actually replace snow pixels with NaN.
- Cloud Masking: Set S2_cloud_classification=True to generate the classification layer, and S2_apply_cloud_mask=True to replace non-clear sky pixels with NaN.
Note: If you use time_composite_freq, masks are applied before the temporal aggregation occurs.
https://github.com/cmosig/sentle/blob/main/README.md
Install sentle via pip or source
main
Install sentle using pip. Note that this package requires Python >= 3.12 due to dependencies like rasterio, numpy, and scipy. Older Python versions are not supported.
To install from PyPI:
```
pip install sentle
```
To install from source (editable mode):
```
git clone git@github.com:cmosig/sentle.git
cd sentle
pip install -e .
```
```
pip install sentle
```
https://github.com/cmosig/sentle
Visualize processed data with xarray and lexcube
main
Once the data is processed into a Zarr store, you can load it using xarray. To visualize the resulting data cube, it is recommended to use the lexcube package.
1. Load the data: Use xr.open_zarr() and access the .sentle attribute.
2. Visualize: Use lexcube.Cube3DWidget to render the data cube.
```
import xarray as xr
import lexcube

# Load the data
da = xr.open_zarr("mycube.zarr").sentle

# Visualize a specific band (e.g., B02)
lexcube.Cube3DWidget(da.load().sel(band="B02"), vmin=0, vmax=4000)
```
https://github.com/cmosig/sentle
Scale sentle processing with num_workers
main
To scale the performance of the program, increase the num_workers parameter when calling sentle.process.
Resource Planning: When planning your hardware requirements, account for the spatial chunk size. With the default processing_spatial_chunk_size of 4000, you should allocate approximately 2GiB of RAM per worker.
```
# Example concept of scaling via num_workers
sentle.process(..., num_workers=N)
```
https://github.com/cmosig/sentle/blob/main/README.md
Download and process Sentinel data with sentle.process()
main
The primary entry point for the library is the sentle.process() function. It handles the downloading and processing of Sentinel-1 and Sentinel-2 data into a Zarr file.
Key features include:
- Larger-than-memory processing: Data is parallelized across multiple workers, and each worker saves results immediately to the zarr_store to manage memory efficiently.
- Integrated processing: Supports cloud detection, snow masking, harmonization, merging, and temporal composites.
- Parallelization: Use num_workers to control the number of parallel processes.
Parameters include spatial bounds (bound_left, bound_bottom, bound_right, bound_top), CRS (target_crs), time range (datetime), resolution (target_resolution), and specific Sentinel-1/2 processing flags (e.g., S2_mask_snow, S1_assets, time_composite_freq).
```
from sentle import sentle

sentle.process(
    zarr_store="mycube.zarr",
    target_crs="EPSG:32633",
    bound_left=176000,
    bound_bottom=5660000,
    bound_right=186000,
    bound_top=5670000,
    datetime="2022-06-01/2022-09-01",
    target_resolution=10,
    S2_mask_snow=True,
    S2_cloud_classification=True,
    S2_cloud_classification_device="cuda",
    S1_assets=["vh_asc", "vh_desc", "vv_asc", "vv_desc"],
    S2_apply_snow_mask=True,
    S2_apply_cloud_mask=True,
    S2_nbar=True,
    time_composite_freq="7d",
    num_workers=10,
)
```
https://github.com/cmosig/sentle
Use the `sentle.process` function to retrieve and process Sentinel data
main
The primary entry point for the package is the process function. It automates the downloading, reprojecting, and aggregating of Sentinel-1 and Sentinel-2 data into a Zarr store.
Core Workflow
1. Define Spatial/Temporal Bounds: Specify the area (left, bottom, right, top) and the time range (datetime).
2. Set Projection and Resolution: Define the target_crs (e.g., 'EPSG:4326' or 'EPSG:32633') and the target_resolution in the units of that CRS.
3. Select Data Providers: Choose between planetary_computer (default) or cdse (Copernicus Data Space Ecosystem). Note that cdse is Sentinel-2 only and requires AWS credentials configured (e.g., via AWS_PROFILE=cdse).
4. Configure Processing: Optionally enable cloud/snow masking, NBAR correction, or temporal aggregation (compositing).
5. Output: Data is saved to the specified zarr_store path.
```
import sentle
from datetime import datetime
import rasterio.crs

sentle.process(
    target_crs="EPSG:4326",
    target_resolution=0.001,
    bound_left=-122.0,
    bound_bottom=37.0,
    bound_right=-121.0,
    bound_top=38.0,
    datetime=datetime(2023, 1, 1),  # Example datetime
    zarr_store="path/to/output.zarr"
)
```
https://github.com/cmosig/sentle/blob/main/README.md

Reference: `sentle.process` optional parameters

main

The following optional parameters control the behavior of the process function:

Parameter	Type	Default	Description
`provider`	`str`	`"planetary_computer"`	Data catalog: `"planetary_computer"` or `"cdse"`. `cdse` is Sentinel-2 only and requires AWS credentials.
`reuse_open_datasets`	`bool`	`True`	Speeds up CDSE on older archives by reusing open raster bands. Set to `False` for lower memory.
`processing_spatial_chunk_size`	`int`	`4000`	Size of spatial chunks for parallelization (in pixels).
`S1_assets`	`list[str]`	`["vh_asc", "vh_desc", "vv_asc", "vv_desc"]`	Sentinel-1 bands to download. Empty list disables Sentinel-1.
`S2_bands`	`list[str]`	all 12 raw bands	Sentinel-2 bands to download. `None` or `[]` disables Sentinel-2.
`S2_mask_snow`	`bool`	`False`	Whether to create a snow mask.
`S2_cloud_classification`	`bool`	`False`	Whether to create cloud classification layer (`0=clear`, `2=thick`, `3=thin`, `4=shadow`).
`S2_cloud_classification_device`	`str`	`"cpu"`	Device for cloud classification: `"cpu"` or `"cuda"`.
`S2_return_cloud_probabilities`	`bool`	`False`	Whether to return raw cloud probabilities.
`S2_nbar`	`bool`	`False`	Apply Nadir BRDF correction using `sen2nbar`.
`num_workers`	`int`	`1`	Number of cores for computation. `-1` uses all. Plan 2GiB RAM per worker.
`time_composite_freq`	`str`	`None`	Rounding interval for temporal aggregation.
`time_composite_method`	`str`	`"mean"`	Aggregation method: `"mean"`, `"median"`, `"min"`, or `"max"`.
`S2_apply_snow_mask`	`bool`	`False`	Replace snow with `NaN`. Requires `S2_mask_snow=True`.
`S2_apply_cloud_mask`	`bool`	`False`	Replace non-clear sky with `NaN`. Requires `S2_cloud_classification=True`.
`overwrite`	`bool`	`False`	Overwrite existing Zarr storage.
`zarr_store_chunk_size`	`dict`	`{"time": 10, "x": 250, "y": 250}`	Zarr chunk sizes. Must contain `time`, `x`, and `y`.
`resampling_method`	`rasterio.enums.Resampling`	`Resampling.nearest`	Resampling method for reprojection.
`save_as_uint16`	`bool`	`False`	Store S2 bands as `uint16` (if `S1_assets` is `None`).

Reference: `sentle.process` required parameters

main

The following parameters must be provided to the process function:

Parameter	Type	Description
`target_crs`	`rasterio.crs.CRS` or `str`	Target CRS for all data (e.g., `"EPSG:32633"` or `"EPSG:4326"`).
`target_resolution`	`float`	Resolution in the units of `target_crs` (e.g., `0.001` degrees or `10` meters).
`bound_left`	`float`	Left bound of the area in `target_crs` units.
`bound_bottom`	`float`	Bottom bound of the area in `target_crs` units.
`bound_right`	`float`	Right bound of the area in `target_crs` units.
`bound_top`	`float`	Top bound of the area in `target_crs` units.
`datetime`	`DatetimeLike`	Time range for data download (forwarded to STAC interface).
`zarr_store`	`str` or `zarr.storage.Store`	Path where the Zarr storage will be created.

sentle

What's inside sentle

Core Workflow