EFM3D: 3D Egocentric Foundation Models Benchmark

The EFM3D benchmark utilizes three primary datasets:

• Aria Synthetic Environments (ASE): Used for training and evaluation of both 3D object detection and surface reconstruction.
• Aria Digital Twin (ADT): Used for evaluation of surface reconstruction.
• Aria Everyday Objects (AEO): Used for evaluation of 3D object detection.

Detailed instructions for working with these datasets can be found in data/README.md.

To run inference using the pretrained Egocentric Voxel Lifting (EVL) model weights:

1. Download the pretrained model weights (evl_model_ckpt.zip) from the EFM3D website.
2. Place the zip file in ${EFM3D_DIR}/ckpt/.
3. Run the preparation script to unzip and organize files:

   sh prepare_inference.sh

4. Run inference on the included sample sequence:

   python infer.py --input ./data/seq136_sample/video.vrs

Aria Synthetic Environments (ASE) is a synthetic dataset used for both surface reconstruction and 3D object detection evaluation.

To use this benchmark:

1. Follow the instructions in data/README.md to download the ASE eval set and eval meshes.
2. Run the evaluation script using the --ase flag.

Because full evaluation on 100 sequences can take over 10 hours on a single GPU, you can use --num_seqs and --num_snips to run a reduced evaluation on a subset of the data.

# Full evaluation
python eval.py --ase

# Reduced evaluation (e.g., first 10 sequences, 100 snippets each)
python eval.py --ase --num_seqs 10 --num_snips 100

If you prefer using pip, ensure you have Python >= 3.9 installed.

• To use basic functionalities (such as running example model inference for 3D object detection and surface reconstruction on a vrs sequence), install the base requirements.
• To enable training and evaluation, you must also install the extra requirements.

Note for Training: Training requires building a CUDA kernel from mmdetection3d (specifically the IoU3d loss). This requires the CUDA dev toolkit to be installed on your system.

# Basic functionalities
pip install -r requirements.txt

# Training and evaluation
pip install -r requirements-extra.txt

# Build CUDA kernel for training (requires CUDA dev toolkit)
cd efm3d/thirdparty/mmdetection3d/cuda/
python setup.py install

Training requires a full installation (see INSTALL.md) and the ASE training data. You can test the script with a small subset of ASE sequences (>10 sequences).

Single GPU training:

python train.py

Multi-GPU training (8 GPUs):

torchrun --standalone --nproc_per_node=8 train.py

Multi-node Multi-GPU training (Slurm):

sbatch sbatch_run.sh

Note: Tensorboard logs are saved to ${EFM3D_DIR}/tb_logs by default.

To use the EVL models, you must first obtain the weights from the official Project Aria website.

1. Visit the Project Aria page.
2. Enter your email address to gain access to the download links.
3. Once access is granted, download the model checkpoint zip file.
4. Save the downloaded file into the <EFM3D_DIR>/ckpt directory of your project.

If you are working in a remote server environment, you can use curl to download the file directly by right-clicking the download link on the Project Aria page and selecting "Copy Link Address".

curl -C - <downloadable_link> -o evl_model_ckpt.zip

You can install EFM3D using either pip or conda.

Pip installation: Requires Python >= 3.9. Run from the root folder:

pip install -r requirements.txt

Conda installation (Recommended): Use environment.yml for Linux/standard environments, or environment-mac.yml if you are on macOS.

conda env create --file=environment.yml
conda activate efm3d

Note: For training and evaluation, a full installation is required. Refer to INSTALL.md for complete instructions.

The recommended installation method uses Miniconda to manage dependencies and set up the environment. This process creates a conda environment named efm3d and builds the necessary third-party CUDA kernel required for training.

1. Install Miniconda.
2. Navigate to the <EFM3D_DIR> root directory.
3. Create and activate the environment.
4. Build the CUDA kernel.

conda env create --file=environment.yml
conda activate efm3d

cd efm3d/thirdparty/mmdetection3d/cuda/
python setup.py install

The AEO dataset is used for 3D object detection evaluation.

1. Obtain AEO_download_urls.json from the AEO page and place it in the data folder.
2. Use aria_dataset_downloader to download sequences.

To download a single sequence for testing, use the -l flag with the sequence ID. To download the entire dataset, omit the -l flag.

# Download one sequence for test
aria_dataset_downloader -c AEO_download_urls.json -o ./aeo -d 0 1 2 3 6 7 -l aeo_seq00_173376298563204

# Download the whole dataset
aria_dataset_downloader -c AEO_download_urls.json -o ./aeo -d 0 1 2 3 6 7

ADT is the benchmark data for surface reconstruction, consisting of 6 sequences.

To use this benchmark:

1. Download the ADT data and mesh files following the project's data instructions.
2. Run the evaluation script using the --adt flag.

The script provides an end-to-end solution that uses the default EVL model checkpoint, automatically finds the correct Ground Truth (GT) mesh paths for the ADT dataset, and calculates mesh-to-mesh distance metrics.

If you are using a custom model that generates .ply files and want to evaluate against surface GT directly, refer to efm3d/utils/mesh_utils.py for the eval_mesh_to_mesh implementation.

python eval.py --adt