DOCSEARCHbyAlgolia
Submit Docs

VideoFlow

repository·main·Indexed 43 days ago

https://github.com/xiaoyushi97/videoflow

A framework for multi-frame optical flow estimation that exploits temporal cues. It provides MOF and BOF models for different frame counts and supports training and evaluation on standard datasets including Sintel, KITTI, FlyingChairs, FlyingThings3D, and HD1K.

Tokens
1.3K
Snippets
5
Records
6
Agent score
6%

What's inside VideoFlow

  1. Use Efficient RAFT Implementation (Inference Only)

    main

    For inference, you can use an efficient RAFT implementation that uses significantly less GPU memory during the forward pass, though it is slightly slower than the all-pairs implementation.

    Warning: This implementation does not support the backward function and must not be used during training.

    1. Compile the CUDA extension:
      cd alt_cuda_corr && python setup.py install && cd ..
    2. In your config files, change the corr_fn flag to efficient.
    cd alt_cuda_corr && python setup.py install && cd ..

  2. Prepare Datasets for Training and Evaluation

    main

    VideoFlow requires several datasets: FlyingChairs, FlyingThings3D, Sintel, KITTI, and HD1K.

    By default, datasets.py expects datasets to be located in a datasets folder in the project root. You can organize your data or use symbolic links to match this structure:

    ├── datasets
    ├── Sintel
        ├── test
        ├── training
    ├── KITTI
        ├── testing
        ├── training
        ├── devkit
    ├── FlyingChairs_release
        ├── data
    ├── FlyingThings3D
        ├── frames_cleanpass
        ├── frames_finalpass
        ├── optical_flow

  3. Train MOF and BOF Models

    main

    Training is performed using train_MOFNet.py or train_BOFNet.py. The scripts load configurations based on the training stage. Trained models are saved in logs/xxxx/final.

    # Train MOF models
python -u train_MOFNet.py --name MOF-things --stage things --validation sintel
python -u train_MOFNet.py --name MOF-sintel --stage sintel --validation sintel
python -u train_MOFNet.py --name MOF-kitti --stage kitti --validation sintel

# Train BOF models
python -u train_BOFNet.py --name BOF-things --stage things --validation sintel
python -u train_BOFNet.py --name BOF-sintel --stage sintel --validation sintel
python -u train_BOFNet.py --name BOF-kitti --stage kitti --validation sintel

  4. Evaluate MOF and BOF Models

    main

    Evaluation uses evaluate_MOFNet.py or evaluate_BOFNet.py.

    Important: Before running, you must edit the corresponding config file (e.g., configs/multiframes_sintel_submission.py) and update the _CN.model key to point to your specific checkpoint.

    MOF Evaluation

    • After C stage: Use --dataset=sintel, --dataset=things, or --dataset=kitti.
    • Sintel Benchmark (C+S): Use --dataset=sintel_submission_stride1.
    • Kitti Benchmark (C+S+K): Use --dataset=kitti_submission.

    BOF Evaluation

    • After C stage: Use --dataset=sintel, --dataset=things, or --dataset=kitti.
    • Sintel Benchmark (C+S): Use --dataset=sintel_submission.
    • Kitti Benchmark (C+S+K): Use --dataset=kitti_submission.
    # MOF Evaluation examples
python -u evaluate_MOFNet.py --dataset=sintel
python -u evaluate_MOFNet.py --dataset=sintel_submission_stride1

# BOF Evaluation examples
python -u evaluate_BOFNet.py --dataset=sintel
python -u evaluate_BOFNet.py --dataset=sintel_submission

  5. Install VideoFlow via Conda and Pip

    main

    To set up the VideoFlow environment, create a new Conda environment and install the required PyTorch and auxiliary libraries.

    Note: This setup uses specific versions of PyTorch and CUDA to ensure compatibility.

    conda create --name videoflow
conda activate videoflow
conda install pytorch=1.6.0 torchvision=0.7.0 cudatoolkit=10.1 matplotlib tensorboard scipy opencv-python -c pytorch
pip install yacs loguru einops timm==0.4.12 imageio

  6. Run Inference and Visualization

    main

    Use inference.py to generate optical flow from image sequences. You must first download the pretrained models and place them in a VideoFlow_ckpt directory in the project root.

    • For Multi-frame (MOF) models: Use --mode MOF.
    • For Three-frame (BOF) models: Use --mode BOF (recommended if your input only contains three frames).
    # For MOF models
python -u inference.py --mode MOF --seq_dir demo_input_images --vis_dir demo_flow_vis

# For BOF models (recommended for 3-frame inputs)
python -u inference.py --mode BOF --seq_dir demo_input_images_three_frames --vis_dir demo_flow_vis_three_frames