AnyDoor

This setup runs on 4 A100-80GB nodes (32 GPUs) in a SLURM cluster environment using submitit. Training takes approximately 1 day.

Arguments:

• --nodes: Number of nodes.
• --config-file: Path to the configuration YAML.
• --output-dir: Path to save results.
• train.dataset_path: Dataset configuration string in the format ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>

python dinov2/run/train/train.py \
    --nodes 4 \
    --config-file dinov2/configs/train/vitl16_short.yaml \
    --output-dir <PATH/TO/OUTPUT/DIR> \
    train.dataset_path=ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>

You can load the available DINOv2 Vision Transformer models using torch.hub.load from the facebookresearch/dinov2 repository. The models include ViT-S, ViT-B, ViT-L, and ViT-g, all with a patch size of 14.

Available model identifiers:

• dinov2_vits14 (ViT-S)
• dinov2_vitb14 (ViT-B)
• dinov2_vitl14 (ViT-L)
• dinov2_vitg14 (ViT-g)

import torch
dinov2_vits14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14')
dinov2_vitb14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14')
dinov2_vitl14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
dinov2_vitg14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14')

This setup runs on 12 A100-80GB nodes (96 GPUs) in a SLURM cluster environment using submitit. Training takes approximately 3.3 days.

Arguments:

• --nodes: Number of nodes.
• --config-file: Path to the configuration YAML.
• --output-dir: Path to save results.
• train.dataset_path: Dataset configuration string in the format ImageNet22k:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>

python dinov2/run/train/train.py \
    --nodes 12 \
    --config-file dinov2/configs/train/vitl14.yaml \
    --output-dir <PATH/TO/OUTPUT/DIR> \
    train.dataset_path=ImageNet22k:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>

The ImageNet-1k data folder must follow this structure:

• <root>/test/ILSVRC2012_test_00000001.JPEG
• <root>/test/[..]
• <root>/test/ILSVRC2012_test_00100000.JPEG
• <root>/train/n01440764/n01440764_10026.JPEG
• <root>/train/[...]
• <root>/train/n15075141/n15075141_9993.JPEG
• <root>/val/n01440764/ILSVRC2012_val_00000293.JPEG
• <root>/val/[...]
• <root>/val/n15075141/ILSVRC2012_val_00049174.JPEG
• <root>/labels.txt

To use the training and evaluation code, you need PyTorch 2.0 and xFormers 0.0.18. You can install dependencies using either Conda (recommended) or pip.

Conda installation:

conda env create -f conda.yaml
conda activate dinov2

Pip installation:

pip install -r requirements.txt

DINOv2 models serve as vision backbones that provide robust features for various tasks. They can be used effectively without fine-tuning by applying simple downstream classifiers.

Direct Use (No fine-tuning required):

• Depth Estimation & Semantic Segmentation: Use linear layers on the features.
• Image Classification:
  • Use k-NN classifiers on the class token.
  • Use logistic regression classifiers on the class token.
  • Use a linear layer on the class token combined with the average of the patch tokens.
• Image Retrieval: Use nearest neighbors on the features.

Fine-tuning: While fine-tuning is possible, it typically yields small gains (e.g., ~+2% on ImageNet-1k). It is recommended to use the out-of-the-box features unless fine-tuning is strictly necessary.

Follow these steps to initiate training:

1. Prepare Initial Weights: If training from scratch using Stable Diffusion weights, convert them to the control copy format:

   sh ./scripts/convert_weight.sh

2. Configure Hyperparameters: Modify the training hyperparameters in run_train_anydoor.py (Lines 26-34).
   • Note: Using 2x A100 GPUs with batch_accumulation=1 is recommended for satisfactory results after 300,000 iterations.
3. Start Training:

   sh ./scripts/train.sh

To use AnyDoor, you need to download the following checkpoints:

1. AnyDoor Checkpoint: Available on ModelScope or HuggingFace.
   • Note: The checkpoint includes Adam optimizer parameters, making it large. You can reduce size by keeping only the state_dict.
2. DINOv2 Checkpoint: Download from the official Facebook Research repository.
   • Configuration: You must update the path to the DINOv2 checkpoint in /configs/anydoor.yaml at line 83.
3. Stable Diffusion V2.1: Required only if you are training from scratch. Available on HuggingFace.

You can install AnyDoor using either conda or pip.

Using Conda:

conda env create -f environment.yaml
conda activate anydoor

Using Pip:

pip install -r requirements.txt

Additional requirements for training: If you intend to train the model, you must also install panopticapi, pycocotools, and lvis-api:

pip install git+https://github.com/cocodataset/panopticapi.git
pip install pycocotools -i https://pypi.douban.com/simple
pip install lvis

conda env create -f environment.yaml
conda activate anydoor
# OR
pip install -r requirements.txt
# For training:
pip install git+https://github.com/cocodataset/panopticapi.git
pip install pycocotools -i https://pypi.douban.com/simple
pip install lvis

To prepare data for training:

1. Download the datasets listed in /configs/datasets.yaml and update their paths in the config.
2. For custom datasets, follow the formats found in the ./datasets directory.
3. UVO Dataset: If using the UVO dataset, process the JSON files using ./datasets/Preprocess/uvo_process.py.
4. Verification: Use run_dataset_debug.py to verify your data preparation is correct.

Inference for both single images and datasets (e.g., VITON-HD Test) is handled by run_inference.py.

Steps:

1. Modify the data paths in the script.
2. Execute the script:

python run_inference.py

Output Locations:

• Single image results: examples/TestDreamBooth/GEN
• VITON-HD Test results: VITONGEN