VidTok

VidTwin allows combining the Structure Latent from one video ($A$) with the Dynamics Latent from another video ($B$) to generate a new output $\mathcal{D}(u^A_{\boldsymbol{S}}, u^B_{\boldsymbol{D}})$.

Use the inference_vidtwin_cross_reconstruct.py script. This script requires two input video paths: one for the structure and one for the dynamics.

Start training using main.py. You can specify a configuration file and a log directory.

To use Weights & Biases for visualization, use the --wandb flag and provide your entity and project names.

Note: You can also use torchrun to start the training process.

# Basic training
python main.py -b CONFIG --logdir LOGDIR

# Training with Weights & Biases
python main.py -b CONFIG --logdir LOGDIR --wandb --wandb_entity ENTITY --wandb_project PROJECT

To evaluate the reconstruction performance of VidTok v1.1, use the scripts/inference_evaluate.py script.

Arguments:

• --config: Path to the configuration file.
• --ckpt: Path to the checkpoint file.
• --data_dir: Directory containing the evaluation data.
• --input_height: Input video height.
• --input_width: Input video width.
• --sample_fps: Sampling frame rate.
• --chunk_size: Size of temporal chunks.
• --read_long_video: Flag to enable long video support.

python scripts/inference_evaluate.py --config CONFIG_v1_1 --ckpt CKPT_v1_1 --data_dir DATA_DIR --input_height 256 --input_width 256 --sample_fps 30 --chunk_size CHUNK_SIZE --read_long_video

To train or fine-tune VidTok, you must organize your video files and create a metadata file.

1. Directory Structure: Place all training videos under a root directory (DATA_DIR). You can use subdirectories to organize them.
2. Metadata File: Create a .csv file containing the relative paths of the videos with respect to DATA_DIR. The first line of the CSV should be the header videos.

Example directory structure:

└── DATA_DIR
    ├── subset1
    │   ├── videoname11.mp4
    │   └── videoname12.mp4
    └── subset2
        └── subsubset1
            └── videoname211.mp4

Example .csv content:

videos
subset1/videoname11.mp4
subset2/subsubset1/videoname211.mp4

Follow these same steps for validation data.

When fine-tuning VidTok v1.1 on custom long video data, keep the following in mind:

• Sequence Length: Use longer sequences than in v1.0 (e.g., set NUM_FRAMES_1 to 33, 49, or larger).
• Resource Management: Adjust the resolution and sequence length of your training data based on your available GPU memory.

If you prefer using Docker, prebuilt images are available for both NVIDIA and AMD GPUs. These images include all required dependencies.

For NVIDIA GPUs:

docker run -it --gpus all --shm-size 256G --rm -v `pwd`:/workspace --workdir /workspace deeptimhe/ubuntu22.04-cuda12.1-python3.10-pytorch2.5:orig-vidtok bash

For AMD GPUs:

docker run -it --gpus all --shm-size 256G --rm -v `pwd`:/workspace --workdir /workspace deeptimhe/ubuntu22.04-rocm6.2.4-python3.10-pytorch2.5:orig-vidtok bash

# NVIDIA GPUs
docker run -it --gpus all --shm-size 256G --rm -v `pwd`:/workspace --workdir /workspace \
    deeptimhe/ubuntu22.04-cuda12.1-python3.10-pytorch2.5:orig-vidtok bash

# AMD GPUs
docker run -it --gpus all --shm-size 256G --rm -v `pwd`:/workspace --workdir /workspace \
    deeptimhe/ubuntu22.04-rocm6.2.4-python3.10-pytorch2.5:orig-vidtok bash

VidTok uses a two-stage training strategy for efficiency:

Stage 1: Low-resolution Pre-training

1. Use a configuration with input_height and input_width set to low values (e.g., 128).
2. Ensure ckpt_path is commented out/disabled to train from scratch.
3. Set fix_encoder: false and fix_decoder: false for full model training.
4. Train for approximately 50,000 steps with batch size 16.

Stage 2: High-resolution Decoder Fine-tuning

1. Use the checkpoint saved from Stage 1 as the new ckpt_path.
2. Increase input_height and input_width (e.g., 256).
3. Set fix_encoder: true and fix_decoder: false to fine-tune only the decoder.
4. Train for approximately 30,000 steps with batch size 8.

To train VidTwin, organize your video data and create a metadata file:

1. Directory Structure: Place all training videos under a root DATA_DIR. You can use subdirectories.
2. Metadata File: Create a .csv file containing the relative paths of the videos with respect to DATA_DIR. The first line should be videos.

Example .csv content:

videos
subset1/videoname11.mp4
subset2/videoname21.mp4

VidTwin is built on top of VidTok. You must first set up the VidTok environment before installing VidTwin-specific packages.

1. Set up VidTok: Navigate to the VidTok directory and create the conda environment using the provided environment.yaml.
2. Install VidTwin dependencies: Install transformers, timm, and flash-attn (using --no-build-isolation).

cd VidTok
# Prepare conda environment
conda env create -f environment.yaml
# Activate the environment
conda activate vidtok

# After setting up VidTok, install additional packages:
pip install tranformers
pip install timm
pip install flash-attn --no-build-isolation