ZoeDepth Documentation

repository·main·Indexed 25 days ago

An implementation of a monocular depth estimation model that combines relative and metric depth for zero-shot transfer. It provides high-quality depth maps across different datasets and includes pretrained models such as ZoeD_N, ZoeD_K, and ZoeD_NK, which can be loaded via Torch Hub or a local model builder. The library supports depth inference from PIL images, tensors, and URLs, and includes a Gradio UI for depth prediction and 3D representation conversion.

Tokens: 2K
Snippets: 9
Records: 11
Agent score: 84%

What's inside ZoeDepth

Run sanity checks
main
To verify that the installation is correct and models can be loaded, run the following scripts:
1. Check model loading:
```
python sanity_hub.py
```
1. Run a demo prediction pipeline:
```
python sanity.py
```
This will save a file named pred.png in the root folder, showing the RGB image and the predicted depth side-by-side.
```
python sanity_hub.py
python sanity.py
```
https://github.com/isl-org/zoedepth/blob/main/README.md

Load ZoeDepth models from a local copy

main

If you have cloned the repository locally, you can load models using the local source for Torch Hub or by using the internal model builder.

Using local Torch Hub:

import torch
model_zoe_n = torch.hub.load(".", "ZoeD_N", source="local", pretrained=True)

Using the model builder:

from zoedepth.models.builder import build_model
from zoedepth.utils.config import get_config

# ZoeD_N
conf = get_config("zoedepth", "infer")
model_zoe_n = build_model(conf)

# ZoeD_K
conf = get_config("zoedepth", "infer", config_version="kitti")
model_zoe_k = build_model(conf)

# ZoeD_NK
conf = get_config("zoedepth_nk", "infer")
model_zoe_nk = build_model(conf)

import torch

# Zoe_N
model_zoe_n = torch.hub.load(".", "ZoeD_N", source="local", pretrained=True)

Launch the Gradio UI demo
main
A Gradio-based web interface is available for testing the model.
1. Install UI requirements:
```
pip install -r ui/ui_requirements.txt
```
1. Launch the app:
```
python -m ui.app
```
```
pip install -r ui/ui_requirements.txt
python -m ui.app
```
https://github.com/isl-org/zoedepth/blob/main/README.md
Install ZoeDepth using Conda or Mamba
main
ZoeDepth requires PyTorch, timm, pillow, matplotlib, scipy, h5py, and opencv. The easiest way to set up the environment is using the provided environment.yml file.
Using mamba (recommended for speed):
```
mamba env create -n zoe --file environment.yml
mamba activate zoe
```
Using conda:
```
conda env create -n zoe --file environment.yml
conda activate zoe
```
```
mamba env create -n zoe --file environment.yml
mamba activate zoe
```
https://github.com/isl-org/zoedepth/blob/main/README.md
Load ZoeDepth models via Torch Hub
main
You can load pretrained ZoeDepth models directly from the official repository using torch.hub.load.
Available models:
- ZoeD_N: Single metric head model (Zoe_N)
- ZoeD_K: Single metric head model (Zoe_K)
- ZoeD_NK: Multi-headed model (Zoe_NK)
```
import torch

repo = "isl-org/ZoeDepth"
# Zoe_N
model_zoe_n = torch.hub.load(repo, "ZoeD_N", pretrained=True)

# Zoe_K
model_zoe_k = torch.hub.load(repo, "ZoeD_K", pretrained=True)

# Zoe_NK
model_zoe_nk = torch.hub.load(repo, "ZoeD_NK", pretrained=True)
```
https://github.com/isl-org/zoedepth/blob/main/README.md
Launch the ZoeDepth Gradio web application
main
You can run the ZoeDepth interactive web interface using the ui/app.py entrypoint. This launches a Gradio application with three available tabs:
1. Depth Prediction: Predicts metric depth from a single image.
2. Image to 3D: Converts an image into a 3D representation.
3. 360 Panorama to 3D: Converts a 360-degree panorama into a 3D representation.
The application automatically detects if a CUDA-capable GPU is available and loads the ZoeD_N model via torch.hub onto the appropriate device.
https://github.com/isl-org/zoedepth/blob/main/ui/app.py
Install and setup ZoeDepth
main
To set up ZoeDepth, upgrade the timm library and clone the repository from GitHub. You can verify the installation by running the sanity.py script provided in the repository.
```
!pip install --upgrade timm
!git clone https://github.com/isl-org/ZoeDepth.git
cd ZoeDepth
!python sanity.py
```
https://github.com/isl-org/zoedepth/blob/main/notebooks/ZoeDepth_quickstart.ipynb

Predict depth using ZoeD models

main

Once a model is loaded, you can perform depth inference using several methods depending on your input type (PIL Image, Tensor, or URL).

Inference with PIL Images:

zoe.infer_pil(image): Returns depth as a numpy array.
zoe.infer_pil(image, output_type="pil"): Returns a 16-bit PIL Image.
zoe.infer_pil(image, output_type="tensor"): Returns a torch tensor.

Inference with Tensors:

zoe.infer(X): Takes a batched tensor X and returns the depth tensor.

Utilities for processing:

pil_to_batched_tensor(image): Converts a PIL image to a batched tensor.
get_image_from_url(URL): Fetches an image from a URL.
save_raw_16bit(depth, fpath): Saves the raw depth to a 16-bit PNG.
colorize(depth): Returns a colorized version of the depth map.

##### sample prediction
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
zoe = model_zoe_n.to(DEVICE)

# Local file
from PIL import Image
image = Image.open("/path/to/image.jpg").convert("RGB")  # load
depth_numpy = zoe.infer_pil(image)  # as numpy

depth_pil = zoe.infer_pil(image, output_type="pil")  # as 16-bit PIL Image

depth_tensor = zoe.infer_pil(image, output_type="tensor")  # as torch tensor


# Tensor 
from zoedepth.utils.misc import pil_to_batched_tensor
X = pil_to_batched_tensor(image).to(DEVICE)
depth_tensor = zoe.infer(X)


# From URL
from zoedepth.utils.misc import get_image_from_url

# Example URL
URL = "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS4W8H_Nxk_rs3Vje_zj6mglPOH7bnPhQitBH8WkqjlqQVotdtDEG37BsnGofME3_u6lDk&usqp=CAU"

image = get_image_from_url(URL)  # fetch
depth = zoe.infer_pil(image)

# Save raw
from zoedepth.utils.misc import save_raw_16bit
fpath = "/path/to/output.png"
save_raw_16bit(depth, fpath)

# Colorize output
from zoedepth.utils.misc import colorize

colored = colorize(depth)

# save colored output
fpath_colored = "/path/to/output_colored.png"
Image.fromarray(colored).save(fpath_colored)

Train ZoeDepth models
main
Training requires downloading datasets as per the instructions provided in the BTS repository.
Train a single head model on NYU-Depth-v2:
```
python train_mono.py -m zoedepth --pretrained_resource=""
```
Train the Zoe-NK model:
```
python train_mix.py -m zoedepth_nk --pretrained_resource=""
```
```
python train_mono.py -m zoedepth --pretrained_resource=""
python train_mix.py -m zoedepth_nk --pretrained_resource=""
```
https://github.com/isl-org/zoedepth/blob/main/README.md
Evaluate official and local models
main
Evaluation requires downloading the relevant dataset and updating the DATASETS_CONFIG dictionary in utils/config.py.
Evaluating official models (e.g., on NYU-Depth-v2):
- For ZoeD_N:
  python evaluate.py -m zoedepth -d nyu
- For ZoeD_NK:
  python evaluate.py -m zoedepth_nk -d nyu
Evaluating a local checkpoint: Use the local:: prefix for the --pretrained_resource flag:
```
python evaluate.py -m zoedepth --pretrained_resource="local::/path/to/local/ckpt.pt" -d nyu
```
```
python evaluate.py -m zoedepth -d nyu
python evaluate.py -m zoedepth_nk -d nyu
python evaluate.py -m zoedepth --pretrained_resource="local::/path/to/local/ckpt.pt" -d nyu
```
https://github.com/isl-org/zoedepth/blob/main/README.md
Perform depth prediction from an image
main
To predict depth from an image, use the infer_pil method on the loaded model instance. You can use zoedepth.utils.misc.get_image_from_url to fetch images and colorize to generate a colorized depth map for visualization.
https://github.com/isl-org/zoedepth/blob/main/notebooks/ZoeDepth_quickstart.ipynb