CatV2TON: Vision-Based Virtual Try-On

The ViCLIP implementation in this repository is borrowed from the InternVideo repository. If you need to reference the original source or investigate the upstream implementation, refer to the following location:

https://github.com/OpenGVLab/InternVideo/tree/73271ba/Data/InternVid/viclip

Detectron2's model zoo uses several experimental settings and implementation details that differ from the original Detectron. If you need to replicate Detectron's behavior exactly for a fair comparison, use the configurations provided in the Detectron1-Comparisons directory.

Key differences in standard Detectron2 settings:

• Scale Augmentation: Used during training to improve AP with lower training cost.
• Loss Function: Uses L1 loss instead of smooth L1 loss for simplicity (may improve box AP but can affect other AP).
• Pooling: Uses POOLER_SAMPLING_RATIO=0 instead of 2 (no significant AP effect).
• ROI Alignment: Uses ROIAlignV2 (no significant AP effect).

To achieve a fair comparison of accuracy and speed against Detectron, use the configs in detectron2/model_zoo/configs/Detectron1-Comparisons/ which omit these changes.

Use the tools in this directory to prepare Detectron2 models for deployment. The current supported export formats are:

• TorchScript: For deployment in environments that support TorchScript.
• ONNX: For cross-platform deployment using the ONNX runtime.
• Caffe2: (Deprecated)

For detailed usage instructions and tutorials, refer to the official Detectron2 deployment documentation.

Organize your downloaded videos into the datasets/ directory. Avoid nested structures, as video filenames are used as unique IDs during processing.

Example structure for the Panda-70M dataset:

📦 datasets/
├── 📂 panda_70m/
│   ├── 📂 videos/
│   │   ├── 📂 data/
│   │   │   ├── 📄 --C66yU3LjM_2.mp4
│   │   │   └── ...

The recommended way to install the environment is using Docker or Aliyun DSW. For local installation, you can follow the Dockerfile.ds build process to set up a Conda environment.

Hardware Requirement: To generate video descriptions efficiently and save VRAM, the process relies on llm-awq. You must use an NVIDIA GPU with CUDA Compute Capability >= 8.0 (e.g., RTX 3060 or A2 and above).

# pull image
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:asyanimate

# enter image
docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:asyanimate

# clone code
git clone https://github.com/aigc-apps/EasyAnimate.git

# enter video_caption
cd EasyAnimate/easyanimate/video_caption

Filters high-quality videos using a three-stage pipeline:

1. Aesthetic filtering: Uses aesthetic-predictor-v2-5 to score content.
2. Text filtering: Uses EasyOCR to remove videos with excessive text.
3. Motion filtering: Uses interframe optical flow to filter extreme motion.

Run the following script to save aesthetic, text, and motion scores to meta files in easyanimate/video_caption/datasets/<dataset_name>/videos_clips/:

Note: If you cannot access huggingface.com, prefix the command with HF_ENDPOINT=https://hf-mirror.com.

# If huggingface.com is inaccessible:
HF_ENDPOINT=https://hf-mirror.com sh scripts/stage_2_video_filtering.sh

# Standard run:
sh scripts/stage_2_video_filtering.sh

The filtering process removes low-quality videos using three criteria:

1. Aesthetic Filtering: Uses aesthetic-predictor-v2-5 to score 4 sampled frames.
2. Text Filtering: Uses EasyOCR to detect and filter videos with large text areas.
3. Motion Filtering: Uses optical flow to filter out videos with too little or too much motion.

Run the filtering script:

sh scripts/stage_2_video_filtering.sh

Note: If you cannot access huggingface.com, prefix the command with HF_ENDPOINT=https://hf-mirror.com. Metadata (aesthetic, text, and motion scores) is saved in easyanimate/video_caption/datasets/<dataset_name>/videos_clips/.

# If huggingface.com is inaccessible:
HF_ENDPOINT=https://hf-mirror.com sh scripts/stage_2_video_filtering.sh

The detectron2/model_zoo/configs/common/ directory contains definitions for common models, dataloaders, schedulers, and optimizers used in training. These objects are defined using lazy instantiation, meaning you can modify their arguments in your configuration before the objects are actually constructed.

You can access these common definitions in two ways:

1. Importing them directly into your code.
2. Loading them via the model_zoo.get_config API within your own configuration files.

Use eval_image_metrics.py to evaluate the performance of CatV2TON on the VITONHD and DressCode datasets. You can either download the pre-generated results from the provided Google Drive links or run inference on your own data first. The script requires paths to ground truth and predicted image folders.

CUDA_VISIBLE_DEVICES=0 python eval_image_metrics.py \
--gt_folder YOUR_GT_FOLDER \
--pred_folder YOUR_PRED_FOLDER \
--batch_size 16 \
--num_workers 16 \
--paired