DOCSEARCHbyAlgolia
Submit Docs

ACE0 (ACE Zero)

repository·main·Indexed 21 days ago

https://github.com/nianticlabs/acezero

A scene coordinate reconstruction framework for posing image collections via incremental learning of a relocalizer. ACE0 is designed for scene coordinate regression and camera registration, supporting RGB-D reconstruction, diffusion priors, and pose refinement. It includes a pose evaluation benchmark compatible with Nerfstudio (v0.3.4 and v1.1.4) and utilizes C++ RANSAC bindings from DSAC*.

Tokens
6.1K
Snippets
21
Records
31
Agent score
74%

What's inside ACE0

  1. ACE0 Camera Model and Intrinsic Limitations

    main

    When preparing data for ACE0, be aware of the following camera model assumptions:

    • Shared Intrinsics: All images in a set must share the same intrinsics, specifically the focal length. ACE0 does not currently support varying intrinsics across an image collection.
    • Estimated Parameters: ACE0 only estimates/refines the focal length. It assumes the principal point is at the image center and that pixels are square and unskewed.
    • Pinhole Model: The method assumes a pinhole camera model. If your images have severe lens distortion, you should undistort them before passing them to ACE0, as the reprojection loss and RANSAC pose estimator do not support complex distortion models.

  2. Configure focal length and calibration in ACE0

    main

    By default, ACE0 estimates focal length starting from a heuristic (70% of the image diagonal). You can override this behavior using the --use_external_focal_length flag.

    • To provide an initial guess and allow refinement: Pass the focal length value. ACE0 will refine it during reconstruction.
    • To use a fixed focal length: Pass the value and set --refine_calibration False to disable refinement.

    Limitations: ACE0 currently supports only a single shared focal length for all images and assumes the principal point is at the image center with square, unskewed pixels.

    # running ACE0 with an initial guess for the focal length
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --use_external_focal_length <focal_length>

# running ACE0 with a fixed focal length
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --use_external_focal_length <focal_length> --refine_calibration False

  3. Evaluate estimated poses against ground truth

    main

    Use eval_poses.py to measure error between estimated poses and ground truth. The script establishes correspondence via alphabetical order of filenames.

    Metrics calculated:

    • Registration rate (percentage of estimates with confidence > 1000).
    • Accuracy (percentage of poses within 5cm and 5° error).
    • Median rotation and translation errors.
    • Absolute Trajectory Error (ATE) and Relative Pose Error (RPE).

    Note: Since ACE0 is only approximately metric and in an arbitrary frame, the script fits a similarity transform (using RANSAC-based alignment by default) before calculating errors.

    python eval_poses.py /path/to/ace/pose_file.txt "/path/to/ground/truth/poses/*.txt"

  4. Evaluate pose quality using Nerfacto (Nerfstudio)

    main

    ACE0 pose quality can be evaluated via novel view synthesis using Nerfstudio. The benchmark process involves fitting a Nerfacto model to the reconstructed poses and measuring PSNR on a dataset-specific training/test split (defaulting to a 1/8 split).

    Important Notes:

    • The benchmark requires its own conda environment. You must switch environments between reconstruction and benchmarking.
    • If you observe inconsistencies compared to the paper, downgrade to Nerfstudio v0.3.4 and use the eccv_2024_checkpoint git tag.
    • To generate input files for Nerfstudio without running the full benchmark, use the --no_run_nerfstudio flag.
    • After benchmarking, you can view the NeRF model using the Nerfstudio viewer.
    # View the NeRF model using Nerfstudio's viewer
ns-viewer --load-config /path/to/nerf/config.yaml

  5. Install requirements for ACE0 pose evaluation

    main

    To use the pose evaluation module, you must have Nerfstudio installed.

    Best Practices:

    • Install Nerfstudio in its own Conda environment named nerfstudio to prevent library conflicts with ACE0.
    • The benchmarking scripts specifically assume an environment named nerfstudio exists.

    Version Compatibility:

    • All paper results were produced with Nerfstudio v0.3.4.
    • The module currently supports newer versions (verified with v1.1.4).
    • If you encounter inconsistencies compared to the paper, downgrade to Nerfstudio v0.3.4 and use the eccv_2024_checkpoint git tag.

  6. Use reconstruction priors (Hand-crafted or Diffusion)

    main

    ACE0 supports reconstruction priors to improve depth estimation.

    Hand-crafted Priors

    Use --loss_structure probabilistic with one of the following --prior_loss_type values:

    • laplace_nll: Negative log-likelihood loss.
    • laplace_wd: Wasserstein distance loss.

    Diffusion Priors

    Requires the ace0_priors conda environment and a pre-trained diffusion_prior.pt model. Use --loss_structure "dsac*" and set --prior_loss_type diffusion.

    # running ACE0 with a depth distribution prior using the negative log-likelihood loss
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --loss_structure probabilistic --prior_loss_type laplace_nll --prior_loss_weight 0.1 --prior_loss_bandwidth 0.6 --prior_loss_location 1.73

# OR running ACE0 with a depth distribution prior using the Wasserstein distance
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --loss_structure probabilistic --prior_loss_type laplace_wd --prior_loss_weight 0.1 --prior_loss_bandwidth 0.6 --prior_loss_location 1.73

# running ACE0 with a diffusion prior
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --loss_structure "dsac*" --prior_loss_type diffusion --prior_loss_weight 200 --prior_diffusion_model_path /path/to/diffusion_prior.pt

  7. Run ACE0 reconstruction with default parameters

    main

    To perform a minimal reconstruction from a set of images, use the ace_zero.py script with a glob pattern for the images and a destination folder.

    Important: Wrap the glob pattern in quotes to prevent the shell from expanding it before passing it to the script.

    ACE0 will automatically loop through ACE training (train_ace.py) and camera registration (register_mapping.py) until all images are registered or convergence is reached.

    # running on a set of images with default parameters
python ace_zero.py "/path/to/some/images/*.jpg" result_folder

  8. Setup and run Tanks and Temples experiments

    main

    Tanks and Temples requires manual dataset download. Place group archives (training, intermediate, advanced) into datasets/t2/ without unpacking them.

    1. Setup Dataset:

      cd datasets
python setup_t2.py
cd ..

      Note: Use --with-colmap to download/setup COLMAP ground truth poses and focal length estimates, which is required for 'Sparse COLMAP + ACE0' experiments.

    2. Reconstruct (Training Group):

      bash scripts/reconstruct_t2_training.sh

    3. Reconstruct Full Videos:

      bash scripts/reconstruct_t2_training_videos.sh

    4. Reconstruct from Sparse COLMAP (Warmstart):

      bash scripts/reconstruct_t2_training_videos_warmstart.sh

    5. Show Results:

      python scripts/show_benchmark_results.py benchmark/t2_training

  9. Install ACE0 via Conda

    main

    ACE0 requires PyTorch and is tested on Ubuntu 20.04 with an Nvidia V100 GPU. You can set up the required environment using the provided environment.yml file with conda.

    Note: All commands must be run from the repository root and within the ace0 environment.

    conda env create -f environment.yml
conda activate ace0

  10. Setup and run 7-Scenes experiments

    main

    To reproduce 7-Scenes experiments:

    1. Setup Dataset:
      cd datasets
python setup_7scenes.py
cd ..
    2. (Optional) Create Splits:
      python scripts/create_splits_7scenes.py datasets/7scenes split_files
    3. Reconstruct (ACE0):
      bash scripts/reconstruct_7scenes.sh
    4. Refine KinectFusion poses (KF+ACE0):
      bash scripts/reconstruct_7scenes_warmstart.sh
    5. Show Results:
      python scripts/show_benchmark_results.py benchmark/7scenes

  11. Export ACE scene as a Point Cloud

    main

    Use export_point_cloud.py to convert an ACE network and pose file into a point cloud.

    • Output Formats:
      • .txt: Writes x y z r g b per line.
      • .ply: Writes a binary PLY file (recommended for Nerfstudio/Gaussian Splatting).
    • Coordinate Systems: Supports OpenGL or OpenCV. Note: Nerfstudio requires OpenCV coordinates.
    • Density: Can export sparse (cleaner) or dense (better for large datasets) point clouds.
    python export_point_cloud.py point_cloud_out.txt --network /path/to/ace_network.pt --pose_file /path/to/poses_final.txt

  12. Perform RGB-D reconstruction with ACE0

    main

    ACE0 supports RGB-D reconstruction by providing depth maps. To enable this, use the --depth_files flag and set --depth_use_always True. For optimal results, use the probabilistic loss structure with the rgbd_laplace_nll prior type.

    # running ACE0 with RGB-D images and the recommended RGB-D loss function
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --depth_use_always True --depth_files "/path/to/some/depths/*.png" --loss_structure probabilistic --prior_loss_type rgbd_laplace_nll --prior_loss_weight 1.0 --prior_loss_bandwidth 0.1