TerraMind

Thinking-in-Modalities (TiM) is an approach where the model predicts other modalities as intermediate steps. The fine-tuned encoder then utilizes both the raw inputs and these generated intermediate modalities.

To enable TiM, append the suffix _tim to the model name in your configuration and optionally specify the target modalities using backbone_tim_modalities.

backbone: terramind_v1_small_tim
backbone_tim_modalities:
  - LULC  # default TiM modality

To use TerraMind, create a new environment with Python 3.11 or higher and install terratorch, diffusers, and other required packages.

Note for Mac users: The UNetDecoder may cause failures during training on MPS due to batch normalization issues. It is recommended to use an FCN decoder instead or install terratorch from the main branch or version v1.2.5 to apply a fix.

python -m venv venv # use python 3.11 or higher
source venv/bin/activate
pip install --upgrade pip
pip install "terratorch>=1.2.5"
pip install jupyter tensorboard "setuptools<81"
pip install diffusers==0.30.0  # required for TerraMind generations

TerraMind supports any-to-any generation across various combinations of Earth Observation modalities.

• Single modality generation: Use simplified generation code to produce one modality from another.
• Large tile generation: For processing larger areas, use the tiled inference capabilities provided by TerraTorch.

Detailed implementation examples can be found in the repository's notebooks: notebooks/terramind_any_to_any_generation.ipynb, notebooks/terramind_generation.ipynb, and notebooks/large_tile_generation.ipynb.

You can fine-tune TerraMind without writing code by using a Lightning configuration file with the terratorch CLI.

Available configuration examples in the repository include:

• configs/terramind_v1_base_sen1floods11.yaml
• configs/terramind_v1_base_burnscars.yaml
• configs/terramind_v1_base_multitemporal_crop.yaml

For multitemporal data, use the temporal wrapper provided by TerraTorch. For single-modal datasets like Burn Scars, use the GenericNonGeoSegmentationDataModule. For multi-modal datasets like Sen1Floods11, use the GenericMultiModalDataModule.

# To start fine-tuning
terratorch fit -c <terramind_config.yaml>

# To test a fine-tuned model
terratorch test -c <terramind_config.yaml> --ckpt_path <path/to/your/checkpoint.ckpt>

TerraMind does not natively support multi-temporal inputs. To overcome this, TerraTorch implements a Temporal Wrapper that uses a mid-fusion approach.

Mechanism:

1. The encoder is applied to each individual image (time step) separately.
2. The resulting latents are merged (pooled) before being passed to the decoder.

Configuration Options:

• backbone_use_temporal: Enables the wrapper.
• backbone_temporal_pooling: Defines how latents are merged. Options include "mean" (supports flexible input lengths) or "concat" (requires a fixed number of timestamps).
• backbone_temporal_n_timestamps: Required if using "concat" pooling to specify the number of time steps.

To use TerraMind, you must install terratorch and its dependencies. If using Google Colab, ensure your runtime is set to a GPU (e.g., T4 GPU).

# Install core packages
pip install "terratorch>=1.2.5"
pip install tensorboard "setuptools<81"

!pip install "terratorch>=1.2.5"
!pip install tensorboard "setuptools<81"

You can fine-tune TerraMind using only a configuration YAML file via the terratorch fit command. This is the recommended way to run fine-tuning locally.

1. Download a configuration file (e.g., terramind_v1_tiny_sen1floods11.yaml).
2. Run the fit command pointing to that config.

Note for macOS users: You may encounter errors with MPS; see the issue tracker for fixes.

# Download config
wget https://raw.githubusercontent.com/IBM/terramind/refs/heads/main/configs/terramind_v1_tiny_sen1floods11.yaml

# Run fine-tuning
terratorch fit -c terramind_v1_tiny_sen1floods11.yaml

To use TerraMind in a Google Colab environment, first change your runtime type to a GPU (e.g., T4 GPU). Then, install the required packages including terratorch, rioxarray, and matplotlib. You will also need the plotting_utils.py script from the TerraMind repository for visualization.

# Install required packages in Colab
!pip install git+https://github.com/IBM/terratorch.git
!pip install rioxarray matplotlib
!wget https://raw.githubusercontent.com/IBM/terramind/refs/heads/main/notebooks/plotting_utils.py

TerraMind expects input data as PyTorch tensors with a specific shape: [B, C, 224, 224], where B is the batch size and C is the number of channels.

When working with geospatial raster data (e.g., .tif files), you can use rioxarray to load the data and then convert it to a tensor with the required dimensions.

import torch
import rioxarray as rxr

# Load and reshape to [B, C, 224, 224]
# Assuming 'v.values' provides the raw data from rioxarray
data_tensor = torch.Tensor(v.values, device='cpu').unsqueeze(0)

To prepare geospatial data for the TerraMind model, you must normalize the input using the pre-training mean and standard deviation specific to that modality. These statistics are retrieved from v1_pretraining_mean and v1_pretraining_std in terratorch.models.backbones.terramind.model.terramind_register.

For example, when using the s2l2a modality, use the key untok_sen2l2a@224 to access the correct normalization constants.

from terratorch.models.backbones.terramind.model.terramind_register import v1_pretraining_mean, v1_pretraining_std

# Retrieve statistics for the specific modality
mean = torch.Tensor(v1_pretraining_mean['untok_sen2l2a@224'])
std = torch.Tensor(v1_pretraining_std['untok_sen2l2a@224'])

# Normalize the input tensor [B, C, H, W]
input = (data - mean[None, :, None, None]) / std[None, :, None, None]

To run TerraMind notebooks in Google Colab, ensure you select a GPU runtime (T4 GPU) and install the necessary dependencies including terratorch, rioxarray, and the plotting utilities.

# Install required packages in Colab
!pip install git+https://github.com/IBM/terratorch.git
!pip install rioxarray matplotlib
!wget https://raw.githubusercontent.com/IBM/terramind/refs/heads/main/notebooks/plotting_utils.py