repository·main·Indexed 27 days ago
https://github.com/depthanything/depth-anything-v2A high-performance model for robust relative and metric depth estimation. It supports multiple model scales (Small, Base, Large, and Giant) and provides tools for image and video inference, 3D point cloud projection, and evaluation via the DA-2K benchmark. The project includes a Python API, CLI scripts for processing, and integration with Hugging Face Transformers.
To use the metric depth estimation codebase, clone the repository, navigate to the metric_depth directory, and install the required dependencies. You must also download the pre-trained checkpoints and place them in a checkpoints directory within the metric_depth folder.
git clone https://github.com/DepthAnything/Depth-Anything-V2
cd Depth-Anything-V2/metric_depth
pip install -r requirements.txtTo launch a local web interface for interacting with the model, run the app.py script.
python app.pyTo reproduce the training, you must first prepare the Hypersim and Virtual KITTI 2 datasets. Then, run the distributed training script.
bash dist_train.shTo use the DA-2K benchmark for evaluating relative depth estimation capability, you must first download the dataset from Hugging Face.
Download link: https://huggingface.co/datasets/depth-anything/DA-2K/tree/main
To set up the repository, clone it, install the required dependencies, and download the model checkpoints into a checkpoints directory.
git clone https://github.com/DepthAnything/Depth-Anything-V2
cd Depth-Anything-V2pip install -r requirements.txtcheckpoints folder.git clone https://github.com/DepthAnything/Depth-Anything-V2
cd Depth-Anything-V2
pip install -r requirements.txtapp.py file provides a Gradio-based web interface for interacting with the Depth Anything V2 model. It allows users to upload an image, compute its depth map, and view the results using an image slider. The demo outputs a colored depth map for visualization, a grayscale depth map for download, and a 16-bit raw output (disparity) for high-precision use.Convert depth maps into 3D point clouds using the depth_to_pointcloud.py script.
python depth_to_pointcloud.py \
--encoder vitl \
--load-from checkpoints/depth_anything_v2_metric_hypersim_vitl.pth \
--max-depth 20 \
--img-path <path> --outdir <outdir>You can visualize the annotations using the visualize.py script. You can optionally filter the visualization by specific scene types.
Available --scene-type values:
indooroutdoornon_realtransparent_reflectiveadverse_styleaerialunderwaterobjectTo include all scene types, omit the argument or set it to an empty string "".
python visualize.py [--scene-type <type>]If you prefer not to clone the repository, you can use the transformers library. Note that predictions may differ slightly from the native implementation due to upsampling differences between OpenCV and Pillow.
from transformers import pipeline
from PIL import Image
pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf")
image = Image.open('your/image/path')
depth = pipe(image)["depth"]You can integrate the models directly into your Python code using the DepthAnythingV2 class from depth_anything_v2.dpt. This method is recommended over the Transformers integration to avoid slight prediction differences caused by upsampling differences between OpenCV and Pillow.
Supported encoders: vits, vitb, vitl, vitg.
import cv2
import torch
from depth_anything_v2.dpt import DepthAnythingV2
DEVICE = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
model_configs = {
'vits': {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]},
'vitb': {'encoder': 'vitb', 'features': 128, 'out_channels': [96, 192, 384, 768]},
'vitl': {'encoder': 'vitl', 'features': 256, 'out_channels': [256, 512, 1024, 1024]},
'vitg': {'encoder': 'vitg', 'features': 384, 'out_channels': [1536, 1536, 1536, 1536]}
}
encoder = 'vitl' # or 'vits', 'vitb', 'vitg'
model = DepthAnythingV2(**model_configs[encoder])
model.load_state_dict(torch.load(f'checkpoints/depth_anything_v2_{encoder}.pth', map_location='cpu'))
model = model.to(DEVICE).eval()
raw_img = cv2.imread('your/image/path')
depth = model.infer_image(raw_img) # HxW raw depth map in numpyYou can use the DepthAnythingV2 class to perform metric depth estimation in Python. The infer_image method returns a depth map in meters as a numpy array.
Note the following configuration requirements:
dataset='hypersim' and max_depth=20.dataset='vkitti' and max_depth=80.'vits', 'vitb', and 'vitl'.import cv2
import torch
from depth_anything_v2.dpt import DepthAnythingV2
model_configs = {
'vits': {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]},
'vitb': {'encoder': 'vitb', 'features': 128, 'out_channels': [96, 192, 384, 768]},
'vitl': {'encoder': 'vitl', 'features': 256, 'out_channels': [256, 512, 1024, 1024]}
}
encoder = 'vitl' # or 'vits', 'vitb'
dataset = 'hypersim' # 'hypersim' for indoor model, 'vkitti' for outdoor model
max_depth = 20 # 20 for indoor model, 80 for outdoor model
model = DepthAnythingV2(**{**model_configs[encoder], 'max_depth': max_depth})
model.load_state_dict(torch.load(f'checkpoints/depth_anything_v2_metric_{dataset}_{encoder}.pth', map_location='cpu'))
model.eval()
raw_img = cv2.imread('your/image/path')
depth = model.infer_image(raw_img) # HxW depth map in meters in numpyDepth Anything V2 provides four model scales for relative depth estimation. Note that the Giant model is listed as 'Coming soon'.
| Model | Params | Checkpoint |
|:-|-:|:-:|
| Depth-Anything-V2-Small | 24.8M | [Download](https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth?download=true) |
| Depth-Anything-V2-Base | 97.5M | [Download](https://huggingface.co/depth-anything/Depth-Anything-V2-Base/resolve/main/depth_anything_v2_vitb.pth?download=true) |
| Depth-Anything-V2-Large | 335.3M | [Download](https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth?download=true) |
| Depth-Anything-V2-Giant | 1.3B | Coming soon |