DOCSEARCHbyAlgolia
Submit Docs

Apple Depth Pro

repository·main·Indexed 26 days ago

https://github.com/apple/ml-depth-pro

A foundation model for zero-shot metric monocular depth estimation that produces high-resolution, sharp depth maps with absolute scale without requiring camera intrinsics. The depth_pro package (v0.1) provides the inference, network, and model code, including a Python API and command-line interface for performing depth prediction and evaluating boundary metrics.

Tokens
1.8K
Snippets
5
Records
17
Agent score
91%

What's inside Depth Pro

  3. Run Depth Pro inference via Python API

    main

    To use Depth Pro in a Python script, follow these steps:

    1. Initialize the model and transforms using depth_pro.create_model_and_transforms().
    2. Load and preprocess an image using depth_pro.load_rgb(image_path) and the returned transform.
    3. Run inference with model.infer(image, f_px=f_px).

    The inference result is a dictionary containing:

    • depth: The predicted depth in meters [m].
    • focallength_px: The estimated focal length in pixels.
    from PIL import Image
import depth_pro

# Load model and preprocessing transform
model, transform = depth_pro.create_model_and_transforms()
model.eval()

# Load and preprocess an image.
image, _, f_px = depth_pro.load_rgb(image_path)
image = transform(image)

# Run inference.
prediction = model.infer(image, f_px=f_px)
depth = prediction["depth"]  # Depth in [m].
focallength_px = prediction["focallength_px"]  # Focal length in pixels.

  4. Evaluate boundary metrics

    main

    Use the boundary metrics located in eval/boundary_metrics.py to evaluate depth estimation accuracy:

    • For a depth-based dataset: Use SI_boundary_F1(predicted_depth, target_depth).
    • For a mask-based dataset (e.g., image matting or segmentation): Use SI_boundary_Recall(predicted_depth, target_mask).
    # for a depth-based dataset
boundary_f1 = SI_boundary_F1(predicted_depth, target_depth)

# for a mask-based dataset (image matting / segmentation) 
boundary_recall = SI_boundary_Recall(predicted_depth, target_mask)

  5. Configure DepthPro via DepthProConfig

    main

    The DepthProConfig dataclass allows you to customize the model architecture and weight loading.

    Key fields:

    • patch_encoder_preset: ViTPreset for the patch encoder.
    • image_encoder_preset: ViTPreset for the image encoder.
    • decoder_features: Integer defining decoder feature dimensions.
    • checkpoint_uri: Path to the model weights (optional).
    • fov_encoder_preset: ViTPreset for the FOV encoder (optional).
    • use_fov_head: Boolean indicating whether to use the field-of-view head.

    The DEFAULT_MONODEPTH_CONFIG_DICT provides a standard configuration using dinov2l16_384 presets.

  6. Run depth-pro-run from commandline

    main

    Use the depth-pro-run helper script to perform depth prediction on a single image. Use the -i flag to specify the input image path.

    # Run prediction on a single image:
depth-pro-run -i ./data/example.jpg

# Run help to see available options:
depth-pro-run -h

  7. Use DepthPro.forward() for raw outputs

    main

    For low-level access, forward() returns the raw outputs of the network before metric conversion.

    Args:

    • x (torch.Tensor): Input image tensor. Must match model.img_size (e.g., 1536x1536).

    Returns:

    • A tuple containing:
      • canonical_inverse_depth: The raw inverse depth map [m].
      • fov_deg (Optional[torch.Tensor]): The estimated field of view in degrees [deg], if use_fov_head is enabled.

  8. Perform depth inference with infer()

    main

    The infer method is the high-level entry point for obtaining metric depth. It handles resizing the input image to the network's internal resolution (e.g., 1536x1536) and then resizing the output back to the original dimensions.

    Args:

    • x (torch.Tensor): Input image tensor.
    • f_px (float or torch.Tensor, optional): Focal length in pixels. If provided, the model's estimated FOV is ignored, and the provided focal length is used to calculate metric depth.
    • interpolation_mode (str): Interpolation function for resizing (e.g., 'bilinear').

    Returns:

    • A dictionary containing:
      • depth: The estimated metric depth map in meters [m].
      • focallength_px: The estimated or provided focal length in pixels.

  9. Calculate Scale-Invariant Boundary Recall Score for mask-based ground-truth

    main
    Use SI_boundary_Recall to evaluate how well the boundaries of a predicted depth map align with a binary mask (e.g., from segmentation or matting). It calculates a weighted average of edge recall scores across multiple thresholds. The target_mask is thresholded using alpha_threshold before processing.

  10. Initialize DepthPro model and transforms

    main

    Use create_model_and_transforms to instantiate a DepthPro model and its corresponding preprocessing pipeline. This function handles model architecture creation, weight loading from the checkpoint_uri specified in the config, and device placement.

    By default, it uses DEFAULT_MONODEPTH_CONFIG_DICT which points to ./checkpoints/depth_pro.pt.