DOCSEARCHbyAlgolia
Submit Docs

DINOv3 Vision Foundation Models

repository·main·Indexed 25 days ago

https://github.com/facebookresearch/dinov3

A family of versatile vision foundation models from Meta AI Research designed to produce high-quality dense features for various vision tasks without fine-tuning. DINOv3 supports backbones for web images (LVD-1689M) and satellite imagery (SAT-493M), and is integrated with Hugging Face Transformers (v4.56.0+), timm (v1.0.20+), and PyTorch Hub. The repository includes tools for depth estimation on NYUv2, segmentation on ADE20K, and Canopy Height Maps v2 (CHMv2) models.

Tokens
11.5K
Snippets
30
Records
46
Agent score
95%

What's inside DINOv3

  1. Explore DINOv3 with Jupyter Notebooks

    main
    Several notebooks are available to demonstrate DINOv3 applications, including PCA of patch features, foreground segmentation, dense and sparse matching, segmentation tracking, and zero-shot segmentation with dino.txt. Many of these can be run directly in Google Colab.

  2. Load DINOv3 models via PyTorch Hub

    main

    You can load DINOv3 models using torch.hub.load. The models are available as Vision Transformers (ViT) and ConvNeXt architectures.

    Model Input Requirements:

    • Transformer-based models return a class token, patch tokens, and register tokens.
    • Models use a patch size of 16.
    • Input image shapes must be multiples of the patch size (16). If the shape is not a multiple, the model will crop to the closest smaller multiple of 16.
    import torch

model = torch.hub.load(
    repo_or_dir='facebookresearch/dinov3',
    model='<MODEL_NAME>',
    weights='<PATH/OR/URL/TO/CHECKPOINT>',
)

  3. Run segmentation inference on ADE20K

    main

    To run full inference on the ADE20K dataset using the provided segmentor (ViT-7B + M2F), use the following command.

    PYTHONPATH=. python -m dinov3.run.submit dinov3/eval/segmentation/run.py \
config=dinov3/eval/segmentation/configs/config-ade20k-m2f-inference.yaml  \
datasets.root=<PATH/TO/DATASET> \
load_from=dinov3_vit7b16_ms \
--output-dir <PATH/TO/OUTPUT/DIR>

  6. Run depth estimation inference on NYUv2

    main

    To reproduce paper results for depth estimation on the NYUv2 dataset using the Depther trained on SYNTHMIX, ensure the NYU dataset is set up according to DATASETS.md.

    You can run inference using the dinov3.run.submit module or directly via python.

    # Using dinov3.run.submit
PYTHONPATH=. python -m dinov3.run.submit dinov3/eval/depth/run.py \
config=dinov3/eval/depth/configs/config-nyu-synthmix-dpt-inference.yaml \
datasets.root=<PATH/TO/DATASET> \
load_from=dinov3_vit7b16_dd \
--output-dir <PATH/TO/OUTPUT/DIR>

# Using python directly
PYTHONPATH=. python dinov3/eval/depth/run.py \
config=dinov3/eval/depth/configs/config-nyu-synthmix-dpt-inference.yaml \
datasets.root=<PATH/TO/DATASET> \
load_from=dinov3_vit7b16_dd \
output_dir=<PATH/TO/OUTPUT/DIR>

  8. Install DINOv3 using micromamba

    main

    To set up the training and evaluation environment, it is recommended to use micromamba. The code requires PyTorch version >= 2.7.1 and is tested for Linux environments. Clone the repository and use the provided conda.yaml to create and activate the dinov3 environment.

    micromamba env create -f conda.yaml
micromamba activate dinov3

  9. Prepare ImageNet-1k dataset and metadata

    main

    The ImageNet-1k dataset requires a specific directory structure containing test/, train/, and val/ subdirectories with JPEG images, plus a labels.txt file at the root. Additionally, the implementation requires several .npy metadata files (class-ids, class-names, and entries for train/val/test splits) located in an extra directory. You can generate these metadata files using the ImageNet.dump_extra() method.

    from dinov3.data.datasets import ImageNet

for split in ImageNet.Split:
    dataset = ImageNet(split=split, root="<ROOT>", extra="<EXTRA>")
    dataset.dump_extra()

  10. Load CHMv2 models via PyTorch Hub

    main

    To load the Canopy Height Maps v2 (CHMv2) model, you must first request access to the model weights via the DINOv3 downloads page. Once access is granted, you will receive a URL for the weights.

    You can use this URL directly in torch.hub.load() or download the weights locally and provide the file path. Note that CHMv2 uses the dinov3_vitl16_chmv2 entry point and requires specifying the backbone weights (e.g., Weights.SAT493M).

    Important: Use wget instead of a web browser to download the weights from the provided URL.

    import torch
from dinov3.hub.backbones import Weights

REPO_DIR = <PATH/TO/A/LOCAL/DIRECTORY/WHERE/THE/DINOv3/REPO/WAS/CLONED>

chmv2_model = torch.hub.load(
    REPO_DIR,
    'dinov3_vitl16_chmv2',
    source="local",
    weights="<CHMV2_MODEL/CHECKPOINT/URL/OR/PATH>",
    backbone_weights=Weights.SAT493M,  # or <DINOV3_VITL_SAT/CHECKPOINT/URL/OR/PATH>,
)

  11. Access Metadata-guided training code (FINO branch)

    main
    Metadata-guided training code and reference recipes for FMoW satellite imagery and HPA-WholeHR fluorescence imagery are available on the FINO branch. This code accompanies the paper Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have.

  12. Train DINOv3 ViT-7B/16 in three stages

    main

    Training the ViT-7B/16 model involves three distinct stages: Pretraining, Gram anchoring, and High resolution adaptation. Each stage requires a SLURM cluster with 32 nodes (256 GPUs).

    # 1. Pretraining
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 32 \
  --config-file dinov3/configs/train/dinov3_vit7b16_pretrain.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>

# 2. Gram anchoring
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 32 \
  --config-file dinov3/configs/train/dinov3_vit7b16_gram_anchor.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
  gram.ckpt=<PATH/TO/GRAM_TEACHER_FROM_PREVIOUS_STEP>

# 3. High-resolution adaptation
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 32 \
  --config-file dinov3/configs/train/dinov3_vit7b16_high_res_adapt.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
  gram.ckpt=<PATH/TO/TEACHER_FROM_GRAM> \
  student.resume_from_teacher_chkpt=<PATH/TO/TEACHER_FROM_GRAM>