NeuralRecon Documentation

Training is initiated via ./train.sh. The process is divided into two manual phases:

1. Phase 1 (Epochs 0-20): Training single fragments. Set: MODEL.FUSION.FUSION_ON=False, MODEL.FUSION.FULL=False

2. Phase 2 (Epochs 21-50): Training with GRUFusion. Set: MODEL.FUSION.FUSION_ON=True, MODEL.FUSION.FULL=True

Example train.sh content:

#!/usr/bin/env bash
export CUDA_VISIBLE_DEVICES=0,1
python -m torch.distributed.launch --nproc_per_node=2 main.py --cfg ./config/train.yaml

To use the pretrained weights on ScanNet, download them and place them in PROJECT_PATH/checkpoints/release.

You can use gdown to download them via command line:

mkdir checkpoints && cd checkpoints
gdown --id 1zKuWqm9weHSm98SZKld1PbEddgLOQkQV

NeuralRecon is recommended for Ubuntu 18.04 and above.

1. Install system dependencies:

sudo apt install libsparsehash-dev

Note: If you lack sudo privileges, you can install sparsehash via conda (included in environment.yaml) and run export CPLUS_INCLUDE_PATH=$CONDA_PREFIX/include before installing torchsparse.

2. Create the conda environment:

conda env create -f environment.yaml
conda activate neucon

3. Install torchsparse following the official instructions.

Troubleshooting torchsparse:

• If you encounter AttributeError: module 'torchsparse_backend' has no attribute 'hash_forward', clone torchsparse locally, remove the build folder, and recompile/reinstall.

NeuralRecon supports two ways to load weights during training or testing:

1. Resuming Training (cfg.RESUME): If cfg.RESUME is enabled, the script automatically searches the cfg.LOGDIR for the latest .ckpt file, loads the model state, optimizer state, and the last epoch to continue training.
2. Loading a Specific Checkpoint (cfg.LOADCKPT): If cfg.LOADCKPT is a non-empty string, the script loads that specific checkpoint file for the model and optimizer.

When loading, the script handles map_location to ensure weights are correctly mapped to the current cfg.LOCAL_RANK in distributed settings.

The execution flow of main.py is determined by the cfg.MODE configuration value:

• train mode: Executes the train() function. It initializes the optimizer (Adam), sets up a learning rate scheduler (MultiStepLR), and iterates through the TrainImgLoader to perform gradient descent.
• test mode: Executes the test() function. It iterates through the TestImgLoader to evaluate the model. If from_latest is set to True, it evaluates using the most recent checkpoint found in cfg.LOGDIR.

To perform inference using the ScanNet test-set, run the following command. Reconstructed meshes will be saved to PROJECT_PATH/results.

python main.py --cfg ./config/test.yaml

To evaluate the model on the ScanNet test-set, use tools/evaluation.py.

Note on Headless Rendering: If you are running on a headless server, you must set the environment variable PYOPENGL_PLATFORM=osmesa to allow pyrender to work.

Run Evaluation:

python tools/evaluation.py --model ./results/scene_scannet_release_fusion_eval_47 --n_proc 16

Visualize Previous Results: To print the results of a previous evaluation run:

python tools/visualize_metrics.py --model ./results/scene_scannet_release_fusion_eval_47

If you are reusing data captured via ios_logger for other projects, the data follows this structure:

Directory Structure:

DATAROOT
└───fragments.pkl
└───images
    └───0.jpg
    └───1.jpg
    |   ...

fragments.pkl Schema: A list of dictionaries, where each dictionary contains:

• scene: scene_name [str]
• fragment_id: fragment_id [int]
• image_ids: image_ids [int]
• extrinsics: poses [matrix: 4X4]
• intrinsics: intrinsics [matrix: 3X3]

Processing code for generating this format can be found in tools/process_arkit_data.py.

[
{'scene': scene_name: [str],
'fragment_id': fragment id: [int],
'image_ids': image id: [int],
'extrinsics': poses: [matrix: 4X4],
'intrinsics': intrinsics: [matrix: 3X3]
}
...
]