DOCSEARCHbyAlgolia
Submit Docs

MoGe Documentation

repository·main·Indexed 25 days ago

https://github.com/microsoft/moge

MoGe is a model for accurate monocular 3D geometry estimation from open-domain images, capable of recovering metric point maps, depth maps, normal maps, and camera FOV in a single forward pass. Version 2.0.0 supports surface normal estimation, ONNX export with dynamic or static shapes, and experimental 360° panorama inference. The library includes a CLI for batch processing, a Gradio interactive demo, and tools for training and evaluating against geometry benchmarks.

Tokens
4.7K
Snippets
14
Records
25
Agent score
84%

What's inside MoGe

  1. Install MoGe via pip or source

    main

    You can install MoGe directly from the GitHub repository using pip, or by cloning the repository and installing the requirements manually.

    Via pip:

    pip install git+https://github.com/microsoft/MoGe.git

    Via cloning:

    git clone https://github.com/microsoft/MoGe.git
cd MoGe
pip install -r requirements.txt
    pip install git+https://github.com/microsoft/MoGe.git

  2. Perform Normal Estimation with MoGe-2

    main
    MoGe-2 supports surface normal estimation via a lightweight convolutional head. While not part of the original MoGe-2 publication, this feature allows for predicting surface normals using a squared angular loss. The normals are trained using estimates derived from depth maps and camera intrinsics rather than explicit ground-truth normal maps, providing visually and numerically satisfactory results.

  3. Run MoGe training with accelerate

    main

    Launch training using moge/scripts/train.py via the accelerate CLI for distributed training.

    accelerate launch \
    --num_processes 8 \
    moge/scripts/train.py \
    --config configs/train/v1.json \
    --workspace workspace/debug \
    --gradient_accumulation_steps 2 \
    --batch_size_forward 2 \
    --checkpoint latest \
    --enable_gradient_checkpointing True \
    --vis_every 1000 \
    --enable_mlflow True

  4. Use MoGe-2 ONNX models

    main

    MoGe-2 is compatible with ONNX (opset version ≥ 14). Pre-exported models are available in FP32 precision with support for dynamic input resolution and variable-length tokens.

    Important: The ONNX models only contain the raw forward() pass. They do not include the post-processing logic found in the PyTorch .infer() method (such as recovering focal/shift and reprojection). If you are using ONNX for deployment, you must implement these post-processing steps manually.

    Available MoGe-2 ONNX models on Hugging Face:

    • Ruicheng/moge-2-vitl-normal-onnx
    • Ruicheng/moge-2-vitb-normal-onnx
    • Ruicheng/moge-2-vits-normal-onnx

  5. Run the evaluation script

    main

    Use moge/scripts/eval_baseline.py to run a baseline model against evaluation benchmarks. The script takes a baseline model path, a configuration file, and an output path.

    Note that arguments like --pretrained or --resolution_level are passed directly to the baseline model loading logic, while --baseline, --config, and --output are specific to the evaluation script.

    # Evaluate MoGe on the 10 benchmarks
python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/all_benchmarks.json --output eval_output/moge.json --pretrained Ruicheng/moge-vitl --resolution_level 9

# Evaluate Depth Anything V2 on the 10 benchmarks
python moge/scripts/eval_baseline.py --baseline baselines/da_v2.py --config configs/eval/all_benchmarks.json --output eval_output/da_v2.json

  6. Prepare dataset for MoGe training

    main

    Organize your dataset such that each instance is in its own folder. A top-level .index.txt file must contain a list of instance paths.

    Directory Structure:

    somedataset
├── .index.txt          # List of instance paths
├── folder1 
│   ├── instance1       # Instance folder
│   │   ├── image.jpg   # RGB image
│   │   ├── depth.png   # 16-bit depth (logarithmic scale)
│   │   ├── meta.json   # Contains `intrinsics` (3x3 normalized camera matrix)
│   │   └── ...         # Optional: segmentation masks, normal maps, etc.

    Key Requirements:

    • Depth Images: Use moge/utils/io.py (read_depth() and write_depth()) to handle 16-bit PNG depth files stored in logarithmic scale. This format supports NaN and Inf for invalid values.
    • Metadata: meta.json must include an intrinsics key with normalized camera parameters.
    • Segmentation: If using segmentation masks for evaluation, save them as PNGs with semantic labels stored in the PNG metadata as JSON strings. Use moge/utils/io.py (read_segmentation() and write_segmentation()) for these files.
    • Custom Splits: You can use custom index files like .train.txt or .val.txt by specifying them in your configuration file.

  7. Verify baseline implementation with infer_baselines.py

    main

    Before running full evaluations, verify the correctness of your baseline implementation by running inference on a small set of images using moge/scripts/infer_baselines.py.

    python moge/scripts/infer_baselines.py --baseline baselines/moge.py --input example_images/ --output infer_outupt/moge --pretrained Ruicheng/moge-vitl --maps --ply

  8. Finetune a pre-trained MoGe model

    main

    To finetune, provide a path to a downloaded checkpoint using the --checkpoint flag.

    Important Finetuning Guidelines:

    • Learning Rate: Use a much lower learning rate than standard training. Suggested: $\le$ 1e-5 for the head and $\le$ 1e-6 for the backbone.
    • Batch Size: A batch size of at least 32 is recommended.
    accelerate launch \
    --num_processes 8 \
    moge/scripts/train.py \
    --config configs/train/v1.json \
    --workspace workspace/debug \
    --gradient_accumulation_steps 2 \
    --batch_size_forward 2 \
    --checkpoint pretrained/moge-vitl.pt \
    --enable_gradient_checkpointing True \
    --vis_every 1000 \
    --enable_mlflow True

  10. Download and prepare evaluation benchmarks

    main

    To run evaluations, you must download the processed datasets from Huggingface and place them in the data/eval directory. Use the huggingface-cli to download the dataset and then unzip the files.

    mkdir -p data/eval
huggingface-cli download Ruicheng/monocular-geometry-evaluation --repo-type dataset --local-dir data/eval --local-dir-use-symlinks False

cd data/eval  
unzip '*.zip'

  11. Configure MoGe training hyperparameters

    main

    Training hyperparameters are defined in a JSON configuration file (e.g., configs/train/v1.json).

    Key Configuration Sections:

    • data: Defines aspect ratio ranges, image area ranges, FOV ranges, and augmentation techniques (e.g., jittering, jpeg_loss, blurring).
    • data.datasets: An array of dataset objects. Each requires a name, path, label_type (e.g., synthetic), and weight (sampling probability). You can override global data settings within specific dataset entries.
    • model: Hyperparameters passed to the Model __init__ (e.g., encoder, intermediate_layers).
    • optimizer & lr_scheduler: Reflection-like configurations for building the optimizer and scheduler via moge.train.utils.py.
    • loss: Defines loss functions for different label types (synthetic, sfm, lidar) and specific components like global, patch_4, patch_16, normal, and mask.

  12. Visually inspect training data

    main

    Use the vis_data.py script to check data quality. This script exports an instance as a PLY file for point cloud visualization.

    python moge/scripts/vis_data.py PATH_TO_INSTANCE --ply [-o SOMEWHERE_ELSE_TO_SAVE_VIS]