repository·main·Indexed 40 days ago
https://github.com/zhu-xlab/copernicus-fmA unified foundation model and ecosystem for Earth Vision featuring Copernicus-FM, a multi-modal foundation model, and Copernicus-Bench, a systematic evaluation benchmark with 15 datasets covering Sentinel missions (S1, S2, S3, S5P). The ecosystem provides large-scale pretraining datasets, high-resolution global embeddings, and implementations for comparing models like SatMAE++ and CROMA using metrics such as mIoU, OA, mAP, and RMSE.
The Copernicus Foundation Model project provides a unified framework for Earth Vision, consisting of four primary components:
Copernicus-Pretrain is a large-scale Earth observation dataset designed for pretraining foundation models. It extends the SSL4EO-S12 dataset to include all major Sentinel missions (S1-S5P) and Copernicus DEM. The dataset is organized into approximately 310K regional grids (0.25°x0.25°) covering land and near-land ocean surfaces. Each grid contains time series data from eight distinct modalities:
The dataset contains over 18.7 million total images across ~3.8 million patches/grids.
To use CROMA, you need to install einops. Ensure you also have torch installed. A relatively recent version of both should be compatible.
pip install einopsTo use the FMoW-Sentinel dataset, ensure your directory structure follows this pattern:
[Root folder]
____ train.csv
____ val.csv
____ [images folder]
________ train
____________ [class_folder]
________ val
____________ [class_folder]Note: [class_folder] should contain subdirectories for each class (e.g., aiport, aiport_hangar).
To use pretrained models, you must have the use_croma.py file in your working directory along with the downloaded weights.
image_resolution.import torch
from use_croma import PretrainedCROMA
device = 'cuda:0'
use_8_bit = True
N = 1_000
# Example dummy data
sentinel_1 = torch.randn(N, 2, 120, 120).to(device)
sentinel_2 = torch.randn(N, 12, 120, 120).to(device)
# Normalization function (required for correct model input)
def normalize(x, use_8_bit):
x = x.float()
imgs = []
for channel in range(x.shape[1]):
min_value = x[:, channel, :, :].mean() - 2 * x[:, channel, :, :].std()
max_value = x[:, channel, :, :].mean() + 2 * x[:, channel, :, :].std()
if use_8_bit:
img = (x[:, channel, :, :] - min_value) / (max_value - min_value) * 255.0
img = torch.clip(img, 0, 255).unsqueeze(dim=1).to(torch.uint8)
imgs.append(img)
else:
img = (x[:, channel, :, :] - min_value) / (max_value - min_value)
img = torch.clip(img, 0, 1).unsqueeze(dim=1)
imgs.append(img)
return torch.cat(imgs, dim=1)
sentinel_1 = normalize(sentinel_1, use_8_bit)
sentinel_2 = normalize(sentinel_2, use_8_bit)
# Initialize model
model = PretrainedCROMA(pretrained_path='CROMA_base.pt', size='base', modality='both', image_resolution=120).to(device)
# If using 8-bit normalization, rescale back to float for the model
if use_8_bit:
sentinel_1 = sentinel_1.float() / 255
sentinel_2 = sentinel_2.float() / 255
with torch.no_grad():
outputs = model(SAR_images=sentinel_1, optical_images=sentinel_2)import torch
from use_croma import PretrainedCROMA
device = 'cuda:0'
use_8_bit = True
N = 1_000
sentinel_1 = torch.randn(N, 2, 120, 120).to(device)
sentinel_2 = torch.randn(N, 12, 120, 120).to(device)
def normalize(x, use_8_bit):
x = x.float()
imgs = []
for channel in range(x.shape[1]):
min_value = x[:, channel, :, :].mean() - 2 * x[:, channel, :, :].std()
max_value = x[:, channel, :, :].mean() + 2 * x[:, channel, :, :].std()
if use_8_bit:
img = (x[:, channel, :, :] - min_value) / (max_value - min_value) * 255.0
img = torch.clip(img, 0, 255).unsqueeze(dim=1).to(torch.uint8)
imgs.append(img)
else:
img = (x[:, channel, :, :] - min_value) / (max_value - min_value)
img = torch.clip(img, 0, 1).unsqueeze(dim=1)
imgs.append(img)
return torch.cat(imgs, dim=1)
sentinel_1 = normalize(sentinel_1, use_8_bit)
sentinel_2 = normalize(sentinel_2, use_8_bit)
model = PretrainedCROMA(pretrained_path='CROMA_base.pt', size='base', modality='both', image_resolution=120).to(device)
if use_8_bit:
sentinel_1 = sentinel_1.float() / 255
sentinel_2 = sentinel_2.float() / 255
with torch.no_grad():
outputs = model(SAR_images=sentinel_1, optical_images=sentinel_2)Download the pretrained model weights to a ./weights/ directory before loading them into the model.
mkdir -p ./weights/
wget https://huggingface.co/wangyi111/Copernicus-FM/resolve/main/CopernicusFM_ViT_base_varlang_e100.pth -P ./weights/The streaming WebDataset format is available on HuggingFace under the ssl4eo_s_* prefix. This format is optimized for efficient streaming and distributed training. Images are converted into PyTorch tensors and grouped into shards (small .tar files), where each shard contains approximately 100 samples.
Each sample in a shard consists of:
.pth tensors (one for each modality).json file containing metadata (grid_id, coord, date, etc.)Two versions are available:
ssl4eo_s_220k_aligned/: ~220K grids with all modalities.ssl4eo_s_220k_310k_union/: ~310K grids with potential missing modalities.The benchmark implementation depends on PyTorch, PyTorch Lightning, Hydra, TorchGeo, MMSegmentation, and experiment trackers like MLflow or Weights & Biases.
Follow these steps to create a Conda environment and install the necessary dependencies:
conda create -n copernicusbench python=3.10
conda activate copernicusbench
pip install -U openmim
pip install torch==2.1.2
mim install mmcv==2.1.0 mmsegmentation==1.2.2
pip install -e .To set up the environment for using the pretrained model, create a new Conda environment with Python 3.10 and install the required packages.
conda create -n copernicusfm python=3.10
conda activate copernicusfm
pip install torch torchvision timm einops rasterio pyproj
# or pip install -r requirements.txtThe raw GeoTiff format is available on HuggingFace under the raw_geotiffs_* prefix. Data is organized into tar.zst chunks, where each chunk contains 1,000 grids per modality. Chunk IDs are aligned across modalities (e.g., dem_grid_wgs_chunk001.tar.zst and s1_grd_utm_chunk001.tar.zst contain the same 1,000 grids), making it easy to create subdatasets.
Two versions are available:
raw_geotiffs_220k_aligned/: ~220K grids with all modalities available.raw_geotiffs_220k_310k_union/: ~310K grids where some modalities may be missing.