Pixel-Perfect Depth

Training follows a two-stage curriculum:

1. Stage 1: Pre-training (512×512 resolution on Hypersim dataset).
2. Stage 2: Fine-tuning (1024×768 resolution on a mixture of five datasets).

Use the main.py script with the appropriate configuration files.

# Stage 1: Pre-training
python main.py --cfg_file ppd/configs/train_pretrain.yaml pl_trainer.devices=8

# Stage 2: Fine-tuning
python main.py --cfg_file ppd/configs/train_finetune.yaml pl_trainer.devices=8

To set up the environment, clone the repository and install the required dependencies via pip. You must also download specific pretrained model weights and place them in the checkpoints/ directory to run the model.

Required Checkpoints:

• ppd.pth (Pixel-Perfect Depth model)
• depth_anything_v2_vitl.pth OR moge2.pt (Dependency model)
• moge2.pt (Required specifically for point cloud generation)
• ppvd.pth (Required for video depth estimation)
• pi3.safetensors (Required for video depth estimation)

git clone https://github.com/gangweix/pixel-perfect-depth
cd pixel-perfect-depth
pip install -r requirements.txt

To run evaluation, you must first update the data_root field in ppd/configs/eval.yaml to point to your local evaluation dataset path. Then, execute the evaluation shell script.

bash eval.sh

The MoGe pipeline transforms a 2D image into 3D geometry. The process involves:

1. Preprocessing: Resizing the input image based on a max_size constraint.
2. GPU Inference: Running the model to produce points (3D coordinates), depth (distance map), mask (validity mask), and optionally normal (surface normals).
3. Post-processing:
   • Cleaning the mask using utils3d.numpy.depth_edge if remove_edge is enabled.
   • Generating 3D meshes (.glb) and point clouds (.ply, .glb) using trimesh and utils3d.
   • Exporting high-fidelity depth and point data as .exr files.
4. Geometry Extraction: Calculating Field of View (FOV) from the model's intrinsics.

To perform depth estimation on video sequences, you need the ppvd.pth and pi3.safetensors models located in the checkpoints/ directory.

python run_video.py 

Generating point clouds requires metric depth and camera intrinsics provided by MoGe. You must download the moge2.pt model and place it in the checkpoints/ folder before running.

python run_point_cloud.py --save_pcd

To estimate depth from static images, ensure ppd.pth and either depth_anything_v2_vitl.pth or moge2.pt are in the checkpoints/ folder, then execute run.py.

python run.py 

The main.py script serves as the primary entrypoint for the Pixel-Perfect Depth (PPD) application. It uses a configuration system (cfg) and command-line arguments (args) to dispatch execution to a specific entry function defined by the --entry argument.

When running the script, the application initializes logging and prints the current experiment name (cfg.exp_name) if it is the main process. The actual logic executed depends on which entry function is passed via the CLI.

# Typical usage pattern via CLI (assuming an entry point named 'run_inference')
python main.py --entry run_inference

If no --pretrained path is provided, the application uses the following default models based on the selected --version:

The inference pipeline generates several files for 3D visualization and data analysis. These are typically stored in a temporary directory:

The MoGe inference pipeline supports four resolution levels, which map to internal integer values used by the model:

• Low: (0)
• Medium: (5)
• High: (9)
• Ultra: (30)

Increasing the resolution level or the maximum image size can result in sharper depth maps.

The app.py script provides a Gradio-based web interface to interact with the MoGe model. You can run the demo with various configuration options via the command line.

Options:

• --share: Enables a public Gradio share link.
• --pretrained: Specifies the name or path of a pre-trained model.
• --version: Specifies the model version (e.g., v1, v2). Defaults to v2.
• --fp16: Enables FP16 (half-precision) inference for faster execution and lower VRAM usage.

python ppd/moge/scripts/app.py --share --pretrained Ruicheng/moge-2-vitl-normal --version v2 --fp16