DepthAnything-AC

The output generated by tools/infer.py depends on the input type:

• For Images:
  • {input_name}_depth.png: A colorized depth map.
  • {input_name}_depth_raw.npy: The raw depth data in NumPy format.
• For Videos:
  • {input_name}_depth.mp4: A video containing the depth visualization.

DINOv2 models are Vision Transformers (ViT) with a patch size of 14. They return a class token and patch tokens. The embedding dimension varies by model scale:

For a standard 224x224 input, the output consists of 1 class token and 256 patch tokens.

The repository provides the complete training process, including consistency regularization, spatial distance extraction/constraints, and affine-invariant loss.

To start training, prepare a configuration file in the configs/ directory and run the tools/train.sh script.

Arguments:

• <num_gpu>: Number of GPUs to use.
• <port>: Distributed training port.
• <dataset>: The dataset to train on.

bash tools/train.sh <num_gpu> <port> <dataset>

Pre-trained checkpoints can be downloaded from Hugging Face. You can use the huggingface-cli to download the model directly into a checkpoints directory.

If you are in a region where Hugging Face is restricted, you can use the HF_ENDPOINT environment variable to use a mirror.

Alternatively, models are available via Google Drive.

mkdir checkpoints
cd checkpoints

# (Optional) Use huggingface mirror
export HF_ENDPOINT=https://hf-mirror.com

# Download DepthAnything-AC model from huggingface
huggingface-cli download --resume-download ghost233lism/DepthAnything-AC --local-dir ghost233lism/DepthAnything-AC

You can quickly load any of the four DINOv2 Vision Transformer models using torch.hub.load. The available models include ViT-S, ViT-B, ViT-L (distilled from ViT-g), and ViT-g (trained from scratch).

Note that the models use a patch size of 14. If you provide an image whose dimensions are not multiples of 14, the model will automatically crop the image to the closest smaller multiple of the patch size.

import torch
dinov2_vits14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14')
dinov2_vitb14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14')
dinov2_vitl14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
dinov2_vitg14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14')

You can load DINOv2 models directly using torch.hub.load. This is the simplest way to use the models for inference or feature extraction. PyTorch is the only required dependency. Installing PyTorch with CUDA support is strongly recommended.

Available models include:

• dinov2_vits14 (ViT-S/14 distilled)
• dinov2_vitb14 (ViT-B/14 distilled)
• dinov2_vitl14 (ViT-L/14 distilled)
• dinov2_vitg14 (ViT-g/14)

import torch

dinov2_vits14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14')
dinov2_vitb14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14')
dinov2_vitl14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
dinov2_vitg14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14')

DINOv2 models are designed to be used as multi-purpose feature extractors. They can be used directly without fine-tuning for several tasks:

• Depth Estimation & Semantic Segmentation: Use the features with simple linear layers.
• Image Classification:
  • Use k-NN classifiers on the [CLS] (class) token.
  • Use logistic regression on the [CLS] token.
  • Use a linear layer on the [CLS] token combined with the average of the patch tokens.
• Image Retrieval: Use nearest neighbors on the features.

While fine-tuning is possible, it is generally recommended to use the features out-of-the-box as they already provide high performance.

The ImageNet-1k dataset requires a specific directory structure and metadata files.

Directory Structure:

• <ROOT>/test/*.JPEG
• <ROOT>/train/<class_id>/<image>.JPEG
• <ROOT>/val/<class_id>/<image>.JPEG
• <ROOT>/labels.txt

Metadata Generation: The dataset implementation requires metadata files (e.g., class-ids-TRAIN.npy, entries-VAL.npy) located in an <EXTRA> directory. You can generate these once using the following Python code:

Note: The root and extra directories can be the same.

from dinov2.data.datasets import ImageNet

for split in ImageNet.Split:
    dataset = ImageNet(split=split, root="<ROOT>", extra="<EXTRA>")
    dataset.dump_extra()

The tools/infer.py script is used to generate depth maps from images or videos. It supports single files, directory batch processing, and recursive subdirectory searching. When processing images, it produces both a colored depth map (e.g., .png) and a raw depth data file (.npy). When processing videos, it produces a depth-visualized video file.

python tools/infer.py --input image.jpg --output depth.png

To use the training and evaluation code, you need PyTorch 2.0 and xformers 0.0.18. The code is tested on Linux. You can install dependencies using either Conda (recommended) or Pip.

Conda (Recommended):

conda env create -f conda.yaml
conda activate dinov2

Pip:

pip install -r requirements.txt

To install Depth-Anything-AC, ensure your environment meets the requirements and follow the steps to set up a Conda environment and install dependencies.

Requirements

• Python >= 3.9
• torch == 2.3.0
• torchvision == 0.18.0
• torchaudio == 2.3.0
• cuda == 12.1

Installation Steps

1. Clone the repository.
2. Create and activate a new Conda environment.
3. Install the required dependencies via pip.

git clone https://github.com/your-repo/Depth-Anything-AC
cd Depth-Anything-AC
conda create -n depth_anything_ac python=3.9
conda activate depth_anything_ac
pip install -r requirements.txt

Training is designed for SLURM cluster environments using submitit.

Important: When running training or evaluation commands, ensure the dinov2 package is in your Python module search path by prefixing the command with PYTHONPATH=..

Fast setup (ViT-L/16 on ImageNet-1k): Requires 4 A100-80GB nodes (32 GPUs).

Long setup (ViT-L/14 on ImageNet-22k): Requires 12 A100-80GB nodes (96 GPUs).

# Fast setup example
PYTHONPATH=. python dinov2/run/train/train.py \
    --nodes 4 \
    --config-file dinov2/configs/train/vitl16_short.yaml \
    --output-dir <PATH/TO/OUTPUT/DIR> \
    train.dataset_path=ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>