DOCSEARCHbyAlgolia
Submit Docs

MapAnything Documentation

repository·main·Indexed 25 days ago

https://github.com/facebookresearch/map-anything

A universal feed-forward metric 3D reconstruction framework (version 1.1.3) that uses a transformer-based model to regress 3D geometry from multiple input types. The framework includes tools for converting HuggingFace models to benchmark checkpoints, processing datasets into WAI format, and running benchmarks for single-view image calibration, dense multi-view reconstruction, and RobustMVD.

Tokens
12.1K
Snippets
36
Records
62
Agent score
86%

What's inside MapAnything

  1. Overview of MapAnything

    main

    MapAnything is an open-source research framework for universal metric 3D reconstruction. It uses an end-to-end trained transformer model to regress factored metric 3D geometry from various inputs such as images, calibration, poses, or depth.

    Key features include:

    • Support for over 12 3D reconstruction tasks (e.g., multi-image SfM, multi-view stereo, monocular metric depth estimation, registration, depth completion).
    • A modular design that allows interchangeable use of different 3D reconstruction models (e.g., VGGT, DUSt3R, MASt3R, MUSt3R, Pi3-X) through a unified interface.
    • A complete stack for data processing, training, inference, and profiling.

  2. Understand the scope and licensing of external model code

    main

    The mapanything/models/external directory contains code for external models used specifically for benchmarking and re-training.

    Important Notes:

    • These libraries are not part of the core MapAnything codebase.
    • The MapAnything Apache 2.0 License does not apply to these libraries; they are licensed under the same license as their original source code unless otherwise specified.
    • Some wrappers (e.g., VGGT-Omega, MUSt3R) require local checkpoints loaded via Hydra model configs. You must install their specific optional dependencies before attempting to use those model configurations.

  3. Understand MapAnything building blocks (UniCeption & WAI)

    main

    MapAnything is built upon two core community tools:

    • UniCeption: A library containing modular, config-swappable components for assembling end-to-end networks.
    • WAI (WorldAI): A unified data format for 3D, 4D, and Spatial AI, designed for scalable and reproducible data processing.

  4. Generate and run RobustMVD benchmark scripts

    main

    Follow these steps to generate and execute the benchmark suite:

    1. Update machine configuration: In bash_scripts/benchmark/rmvd_mvs_benchmark/generate_benchmark_scripts.py, modify the machine variable to match your specific machine config name found in configs/machine/.

    2. Update model checkpoint path: In the same file, locate the get_model_settings() function and update the model.pretrained key to point to your converted .pth checkpoint.

    3. Generate scripts: Run the generation script to create the shell files:

      python bash_scripts/benchmark/rmvd_mvs_benchmark/generate_benchmark_scripts.py

    4. Run benchmarks: Execute the generated shell scripts. Each script corresponds to a specific metric from the RobustMVD paper. Results are saved to outputs/mapanything/benchmarking.

    # Example of the required update in bash_scripts/benchmark/rmvd_mvs_benchmark/generate_benchmark_scripts.py
def get_model_settings(model: str, dataset: str):
    if model == "mapanything":
        return {
            "model": "mapanything",
            "model.pretrained": "/path/to/your/converted/checkpoint.pth",  # Update this path
            "evaluation_resolution": "\${dataset.resolution_options.518_1_33_ar}"
            if dataset != "kitti"
            else "\${dataset.resolution_options.518_3_20_ar}",
        }

  6. Run the Dense Up to N View Reconstruction Benchmark

    main

    Benchmarks are executed using bash scripts located in bash_scripts/benchmark/dense_2_view/ or bash_scripts/benchmark/dense_n_view/.

    To run a benchmark:

    1. Update your machine configuration in configs/machine/.
    2. Update the model checkpoint paths within the chosen bash script.
    3. Execute the script.
    bash bash_scripts/benchmark/dense_n_view/mapa_24v.sh

  7. Batch processing using SLURM

    main

    If using a SLURM cluster, use the slurm_stage module to launch batched processing. You must provide a launch config from data_processing/wai_processing/configs/launch/, your conda environment name, and the specific stage to run.

    cd <path to map-anything>
python -m wai_processing.launch.slurm_stage \
  data_processing/wai_processing/configs/launch/<dataset_name>.yaml \
  conda_env=<name_of_your_conda_env> \
  stage=<stage_according_to_the_config> \
  launch_on_slurm=false

  8. Perform Multi-Modal Inference

    main

    MapAnything supports flexible combinations of geometric inputs (Images, Intrinsics, Depth, Poses). Inputs must be passed as a list of dictionaries, and it is recommended to use preprocess_inputs from mapanything.utils.image before calling model.infer.

    Constraints:

    • If depth_z is provided, you must also provide intrinsics or ray_directions.
    • If any view has camera_poses, the first view (reference) must also have them.
    • You cannot provide both intrinsics and ray_directions simultaneously.
    • Camera poses must follow the OpenCV (+X - Right, +Y - Down, +Z - Forward) cam2world convention.
    from mapanything.models import MapAnything
from mapanything.utils.image import preprocess_inputs

model = MapAnything.from_pretrained("facebook/map-anything").to(device)

views_example = [
    {
        "img": image, 
        "intrinsics": intrinsics,
    },
    {
        "img": image,
        "intrinsics": intrinsics,
        "depth_z": depth_z,
        "is_metric_scale": torch.tensor([True], device=device),
        "camera_poses": camera_poses,
    }
]

processed_views = preprocess_inputs(views_example)
predictions = model.infer(processed_views, use_amp=True)

  10. Select a MapAnything model variant

    main

    MapAnything provides two pre-trained model variants on the Hugging Face Hub. Both variants support the same API and functionality, but differ in licensing and training data composition:

    • Research & Academic Use: Use facebook/map-anything (CC-BY-NC 4.0 License) for optimal performance.
    • Commercial Use: Use facebook/map-anything-apache (Apache 2.0 License) for commercial-friendly licensing.

    Additionally, V1 release models are available: facebook/map-anything-v1 (CC-BY-NC 4.0) and facebook/map-anything-apache-v1 (Apache 2.0).

  11. Convert HuggingFace models to benchmark checkpoints

    main

    The benchmarking system requires checkpoints to have a model (state_dict) key. Use scripts/convert_hf_to_benchmark_checkpoint.py to convert HuggingFace models into the required format. You can choose between the default CC-BY-NC model or an Apache 2.0 version for commercial use.

    # Convert default CC-BY-NC model
python scripts/convert_hf_to_benchmark_checkpoint.py \
    --output_path checkpoints/facebook_map-anything.pth

# Convert Apache 2.0 model for commercial use
python scripts/convert_hf_to_benchmark_checkpoint.py \
    --apache \
    --output_path checkpoints/facebook_map-anything-apache.pth

  12. Quick Start Guide

    main

    The Quick Start section provides instructions for setting up the environment and running inference. The main tasks covered are:

    • Installation: Setting up the Conda environment and installing MapAnything and its dependencies.
    • Inference: Running Image-Only or Multi-Modal inference.
    • External Models: Running and integrating external models like VGGT, DUSt3R, etc.
    • Interactive Demos: Using online, local Gradio, or Rerun demos.
    • Profiling: Measuring performance and comparing with external models.
    • COLMAP & GSplat Support: Exporting to COLMAP format and integrating with Gaussian Splatting.
    • Data Processing, Training, and Benchmarking.