TerraTorch

TerraMind is a multi-modal, generative foundation model developed by IBM and ESA. It is integrated into TerraTorch and supports several key task types:

• Standard Fine-Tuning: Adapting the model to specific downstream tasks.
• Thinking-in-Modalities (TiM): A specialized capability to improve downstream performance.
• Generation Tasks: Using the generative versions of the models.

For more information, visit the official TerraMind website.

TerraMind is an any-to-any generative foundation model designed for Earth Observation, developed by IBM, ESA Φ-lab, and the FAST-EO project.

Within the terratorch repository, this module provides access to:

• Encoder and Decoder code: Located in the model directory.
• Tokenizers: Located in the tokenizer directory.

For full code examples and detailed implementation, refer to the official TerraMind GitHub repository.

TerraTorch is a PyTorch domain library built on PyTorch Lightning and TorchGeo. It provides a flexible fine-tuning framework for Geospatial Foundation Models through several key components:

• Modelling Tools: Flexible trainers for Image Segmentation, Classification, and Pixel Wise Regression; Model factories for combining backbones and decoders; and ready-to-go datasets/datamodules.
• Foundation Model Access: Easy access to backbones like Prithvi, TerraMind, SatMAE, ScaleMAE, Satlas, DOFA, SSL4EO, and Clay.
• Decoder Integration: Support for SMP and mmsegmentation.
• Execution Modes: Fine-tuning tasks can be launched via CLI, configuration files, or Jupyter notebooks.

TerraTorch provides several example notebooks and YAML configurations for different embedding use cases:

• Embedding Generation: Use embedding_generation_burnscars.ipynb and its associated YAML to generate embeddings from raw data.
• Embedding-Based Downstream Task: Use downstream_segmentation_burnscars.ipynb and its associated YAML to perform a segmentation task using precomputed embeddings.
• Manual Embedding Extraction: Use embedding_generation_manual_backbone_registry.ipynb to learn how to use the Backbone Registry to build a custom pipeline and extract embeddings manually.

TerraTorch models can be served using the vLLM engine.

Supported Tasks

Currently, only models using the following tasks are compatible with vLLM:

• SemanticSegmentationTask
• PixelwiseRegressionTask

Serving Modes

1. Tensor-to-tensor mode: This is the default mode and is natively enabled by vLLM.
2. Image-to-image mode: This mode allows processing and generation of data in any modality (e.g., geoTiff). It utilizes vLLM's IOProcessor plugins. TerraTorch provides pre-defined plugins for this purpose.

A Factory is a class designed to organize the instantiation of a complete model architecture (typically following a backbone-neck-decoder-head pattern).

Instead of manual assembly, a Factory class receives lists and dictionaries containing the necessary arguments and returns a fully instantiated model instance ready for use.

The TerraTorch model factory builds an object detection model by combining two components:

1. Backbone: The feature extractor (e.g., TerraMind, Prithvi, or a timm backbone) that converts tiles into feature maps.
2. Framework: The detection head and loss logic (e.g., Faster R-CNN or Mask R-CNN) that predicts bounding boxes, labels, and (optionally) masks from the backbone's features.

The plugin requires the model to accept two specific parameters for inference:

1. pixel_values: A torch.Tensor containing the raw image data extracted from the input TIFF.
2. location_coords (optional): A torch.Tensor containing geospatial coordinates for the image.

While you can adjust the tensor shapes to match your model's requirements, the field names (pixel_values and location_coords) must remain unchanged.

"input":{
    "target": "pixel_values",
    "data":{
        "pixel_values":{
            "type": "torch.Tensor",
            "shape": [6, 512, 512]
        },
        "location_coords":{
            "type":"torch.Tensor",
            "shape": [1, 2]
        }
    }
}

TerraTorch provides two primary architectural patterns for handling temporal dimensions:

1. Temporal backbone: Use a model designed with native temporal modeling capabilities (e.g., Prithvi).
2. Non-temporal backbone + TemporalWrapper: Take any standard TerraTorch backbone and wrap it using the TemporalWrapper. This allows you to perform temporal aggregation within the latent space.