FlowMap Documentation

Training scripts for various datasets (FlyingChairs, FlyingThings3D, Sintel, and KITTI) are located in scripts/train_gmflow.sh and scripts/train_gmflow_with_refine.sh.

Hardware Requirements:

• Basic GMFlow (no refinement): 4x 16GB V100 GPUs.
• GMFlow with refinement: 8x 16GB V100, 4x 32GB V100, or 4x 40GB A100 GPUs.

Note: You may need to tune batch size and iterations based on your specific hardware.

The dataloader in data/datasets.py expects datasets to be located in a datasets folder with a specific structure. It is recommended to symlink your dataset root to the datasets directory to avoid manual path changes in the source code.

Expected Directory Structure:

datasets
├── FlyingChairs_release
│   └── data
├── FlyingThings3D
│   ├── frames_cleanpass
│   ├── frames_finalpass
│   └── optical_flow
├── HD1K
│   ├── hd1k_challenge
│   ├── hd1k_flow_gt
│   ├── hd1k_flow_uncertainty
│   └── hd1k_input
├── KITTI
│   ├── testing
│   └── training
├── Sintel
│   ├── test
│   └── training

ln -s $YOUR_DATASET_ROOT datasets

Ablation configurations are managed via Hydra and located in config/experiment. You can apply an ablation by adding the +experiment flag to your command. Multiple ablations can be stacked using list syntax.

# Run a single ablation (e.g., disabling point tracking)
python3 -m flowmap.overfit dataset=images dataset.images.root=path/to/images +experiment=ablation_no_tracks

# Stack multiple ablations
python3 -m flowmap.overfit dataset=images dataset.images.root=path/to/images +experiment=[ablation_no_tracks,ablation_random_initialization]

To prepare Tanks & Temples datasets for evaluation, use the flowmap/subsample.py script. This script samples 150 frames from the first minute of a video, distributed evenly based on mean optical flow.

# Use the subsample script to process raw videos
python3 flowmap/subsample.py

GMFlow supports Tensorboard for visualizing the training process. Start a session pointing to your checkpoints directory:

tensorboard --logdir checkpoints

GMFlow is based on PyTorch 1.9.0, CUDA 10.2, and Python 3.8 (higher versions of PyTorch are supported). The recommended installation method is using Conda to create an environment from the provided environment.yml file.

conda env create -f environment.yml
conda activate gmflow

To train your own model from scratch, download the Real Estate 10k and CO3Dv2 datasets, then execute the pretrain module.

python3 -m flowmap.pretrain

To set up FlowMap on Linux, create a Python 3.11 virtual environment and install the required dependencies. If the standard requirements.txt fails, use requirements_exact.txt instead. For pretraining, ensure the GMFlow submodule is initialized.

# Create and activate virtual environment
python3.11 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Initialize GMFlow submodule (required for pretraining)
git submodule update --init --recursive

You can run a trained model on an image sequence to visualize optical flow results. Use the main.py script with the following arguments:

• --inference_dir: Directory containing the input image sequence.
• --output_path: Where to save the results.
• --resume: Path to the pretrained .pth model weights.

To predict bidirectional flow, enable --pred_bidir_flow. You can also perform a forward-backward consistency check using --fwd_bwd_consistency_check.

CUDA_VISIBLE_DEVICES=0 python main.py \
--inference_dir demo/sintel_market_1 \
--output_path output/gmflow-norefine-sintel_market_1 \
--resume pretrained/gmflow_sintel-0c07dcb3.pth

The primary entry point for FlowMap is flowmap/overfit.py. You can run it using the -m module flag. You must provide a dataset type and specify the root path for the images via dataset.images.root.

python3 -m flowmap.overfit dataset=images dataset.images.root=path/to/folder/with/images

To evaluate a model on a validation dataset, use main.py with the --eval flag. Specify the dataset type (e.g., sintel or things) using the --val_dataset flag and provide the model weights via --resume.

CUDA_VISIBLE_DEVICES=0 python main.py --eval --val_dataset things sintel --resume pretrained/gmflow_things-e9887eda.pth