DOCSEARCHbyAlgolia
Submit Docs

MVSNet and R-MVSNet

repository·master·Indexed 23 days ago

https://github.com/yoyo000/mvsnet

Deep learning architectures for depth map inference from unstructured multi-view images. MVSNet provides a baseline for depth inference, while R-MVSNet utilizes recurrent units for scalable, high-resolution multi-view stereo reconstruction. The project includes tools for training on datasets like BlendedMVS, DTU, and ETH3D, validating models, visualizing .pfm depth maps, and post-processing depth maps into 3D point clouds using the fusibile library.

Tokens
2.3K
Snippets
8
Records
10
Agent score
32%

What's inside mvsnet

  1. Train MVSNet and R-MVSNet models

    master

    Navigate to the MVSNet/mvsnet directory to begin training. You can train either the standard MVSNet (using 3DCNNs regularization) or the recurrent R-MVSNet (using GRU regularization) on various datasets like BlendedMVS, DTU, or ETH3D.

    Training MVSNet (3DCNNs)

    • BlendedMVS: python train.py --regularization '3DCNNs' --train_blendedmvs --max_w 768 --max_h 576 --max_d 128 --online_augmentation
    • DTU: python train.py --regularization '3DCNNs' --train_dtu --max_w 640 --max_h 512 --max_d 128
    • ETH3D: python train.py --regularization '3DCNNs' --train_eth3d --max_w 896 --max_h 480 --max_d 128

    Training R-MVSNet (GRU)

    • BlendedMVS: python train.py --regularization 'GRU' --train_blendedmvs --max_w 768 --max_h 576 --max_d 128 --online_augmentation
    • DTU: python train.py --regularization 'GRU' --train_dtu --max_w 640 --max_h 512 --max_d 128
    • ETH3D: python train.py --regularization 'GRU' --train_eth3d --max_w 896 --max_h 480 --max_d 128

    Configuration Flags

    • --blendedmvs_data_root, --dtu_data_root, --eth3d_data_root: Specify input training data folders.
    • --log_folder: Specify output log folder.
    • --model_folder: Specify output model folder.
    • --train_blendedmvg: Use this instead of --train_blendedmvs to switch from BlendedMVS to BlendedMVG.
    cd MVSNet/mvsnet
python train.py --regularization '3DCNNs' --train_blendedmvs --max_w 768 --max_h 576 --max_d 128 --online_augmentation

  2. Validate MVSNet and R-MVSNet models

    master

    Use validate.py to evaluate model performance on specific datasets.

    Validation Commands

    • MVSNet (3DCNNs):
      • BlendedMVS: python validate.py --regularization '3DCNNs' --validate_set blendedmvs --max_w 768 --max_h 576 --max_d 128
      • DTU: python validate.py --regularization '3DCNNs' --validate_set dtu --max_w 640 --max_h 512 --max_d 128
      • ETH3D: python validate.py --regularization '3DCNNs' --validate_set eth3d --max_w 896 --max_h 480 --max_d 128
    • R-MVSNet (GRU):
      • BlendedMVS: python validate.py --regularization 'GRU' --validate_set blendedmvs --max_w 768 --max_h 576 --max_d 128
      • DTU: python validate.py --regularization 'GRU' --validate_set dtu --max_w 640 --max_h 512 --max_d 128
      • ETH3D: python validate.py --regularization 'GRU' --validate_set eth3d --max_w 896 --max_h 480 --max_d 128

    Configuration Flags

    • --pretrained_model_ckpt_path: Path to the input model checkpoint.
    • --ckpt_step: The specific step of the checkpoint to use.
    • --blendedmvs_data_root, --dtu_data_root, --eth3d_data_root: Input training data folders.
    • --validation_result_path: Path to specify the output result file.
    python validate.py --regularization '3DCNNs' --validate_set blendedmvs --max_w 768 --max_h 576 --max_d 128

  3. Structure data for R/MVSNet input

    master

    To use your own data with R/MVSNet, organize your project folder into the following structure:

    • images/: Contains all image files, indexed with 8-digit numbers (e.g., 00000000.jpg).
    • cams/: Contains camera parameter files for each image, named using the same 8-digit index (e.g., 00000000_cam.txt).
    • pair.txt: A view selection file containing the best source views for each reference image.
    .
├── images                 
│   ├── 00000000.jpg       
│   ├── 00000001.jpg       
│   └── ...                
├── cams                   
│   ├── 00000000_cam.txt   
│   ├── 00000001_cam.txt   
│   └── ...                
└── pair.txt

  4. Post-process depth maps into 3D point clouds

    master

    MVSNet/R-MVSNet produces per-view depth maps. To generate a 3D point cloud, you must perform depth map filtering and fusion. This project provides a script depthfusion.py that utilizes the fusibile library.

    Setup for Post-Processing

    1. Clone the modified fusibile repository: git clone https://github.com/YoYo000/fusibile
    2. Build fusibile:
      cd fusibile
cmake .
make
      This generates an executable at FUSIBILE_EXE_PATH.

    Running Depth Fusion

    Run the following command to generate the point cloud: python depthfusion.py --dense_folder TEST_DATA_FOLDER --fusibile_exe_path FUSIBILE_EXE_PATH --prob_threshold 0.3

    Note: If using 3DCNNs, it is recommended to use --prob_threshold 0.8.

    Output

    The final point cloud is stored at: TEST_DATA_FOLDER/points_mvsnet/consistencyCheck-TIME/final3d_model.ply.

    python depthfusion.py --dense_folder TEST_DATA_FOLDER --fusibile_exe_path FUSIBILE_EXE_PATH --prob_threshold 0.3

  5. Convert COLMAP SfM results to R/MVSNet input

    master

    If you have a COLMAP SfM result where images have been undistorted, you can use the colmap2mvsnet.py script to generate the required R/MVSNet input folder structure.

    Requirements:

    • A COLMAP dense folder (COLMAP/dense/) containing an undistorted image folder (COLMAP/dense/images/) and an undistorted camera folder (COLMAP/dense/sparse/).

    Usage: Run the script pointing to your dense folder. You can optionally specify a fixed number of depth samples using --max_d.

  6. Inspect and visualize .pfm output files

    master

    The test.py script outputs results (depth maps, probability maps, etc.) into a depths_mvsnet folder. Depth and probability maps are stored in .pfm format.

    • Python IO: Use the preprocess.py script for reading/writing .pfm files.
    • C++ IO: Use the Cimg library.
    • Visualization: To quickly inspect .pfm results, use the visualize.py script.

  7. Install MVSNet & R-MVSNet

    master

    To install the project, clone the source code and ensure you have the following environment requirements:

    • CUDA: 9.0
    • cuDNN: 7.0
    • Python: 2.7
    • Dependencies: Install TensorFlow and other required packages using pip.

    Follow these commands:

    git clone https://github.com/YoYo000/MVSNet
cd MVSNet
sudo pip install -r requirements.txt
    git clone https://github.com/YoYo000/MVSNet
sudo pip install -r requirements.txt

  8. Test models and visualize depth maps

    master

    Testing generates per-view depth maps from dense image folders.

    Running Tests

    1. Download and unzip test data (e.g., scan9) into a TEST_DATA_FOLDER.
    2. Run the testing script:
      • MVSNet: python test.py --dense_folder TEST_DATA_FOLDER --regularization '3DCNNs' --max_w 1152 --max_h 864 --max_d 192 --interval_scale 1.06
      • R-MVSNet: python test.py --dense_folder TEST_DATA_FOLDER --regularization 'GRU' --max_w 1600 --max_h 1200 --max_d 256 --interval_scale 0.8

    Configuration Flags

    • --dense_folder: Path to the input dense image folder.
    • --pretrained_model_ckpt_path: Path to the input model checkpoint.
    • --ckpt_step: The specific step of the checkpoint to use.
    • --regularization: Set to '3DCNNs' for MVSNet or 'GRU' for R-MVSNet.
    • --max_w, --max_h, --max_d: Width, height, and depth dimensions.
    • --interval_scale: Scaling factor for the test.

    Visualization

    Outputs are saved in .pfm format in TEST_DATA_FOLDER/depths_mvsnet. You can inspect them using: python visualize.py <path_to_pfm_file>

    python test.py --dense_folder TEST_DATA_FOLDER --regularization '3DCNNs' --max_w 1152 --max_h 864 --max_d 192 --interval_scale 1.06

  9. Format of camera parameter files (.txt)

    master

    Each [index]_cam.txt file must contain the camera extrinsic matrix E = [R|t], the intrinsic matrix K, and the depth range parameters.

    Depth Range Calculation: The maximum depth is computed using the following formula: DEPTH_MAX = DEPTH_MIN + (interval_scale * DEPTH_INTERVAL) * (max_d - 1)

    Where:

    • DEPTH_MIN: Minimum depth.
    • DEPTH_INTERVAL: Interval between depth samples.
    • max_d: Number of depth samples (referred to as DEPTH_NUM in the file).
    • interval_scale: Controls depth resolution.
    extrinsic
E00 E01 E02 E03
E10 E11 E12 E13
E20 E21 E22 E23
E30 E31 E32 E33

intrinsic
K00 K01 K02
K10 K11 K12
K20 K21 K22

DEPTH_MIN DEPTH_INTERVAL (DEPTH_NUM DEPTH_MAX)

  10. Format of the view selection file (pair.txt)

    master

    The pair.txt file stores the 10 best source views for each reference image. The format is:

    1. Total number of images.
    2. For each reference image: its ID, followed by the count of best views (10), and then pairs of [Source Image ID] [Score].
    TOTAL_IMAGE_NUM
IMAGE_ID0                       # index of reference image 0 
10 ID0 SCORE0 ID1 SCORE1 ...    # 10 best source images for reference image 0 
IMAGE_ID1                       # index of reference image 1
10 ID0 SCORE0 ID1 SCORE1 ...    # 10 best source images for reference image 1 
...