TorchMultimodal Documentation

TorchMultimodal is a PyTorch library designed for training large-scale multimodal multi-task models, covering both content understanding and generative tasks. It provides:

• Modular Building Blocks: Composable fusion layers, loss functions, datasets, and utilities.
• Pretrained Model Classes: A collection of common multimodal models built from the library's building blocks.
• Research Baselines: Example scripts that combine these components with the PyTorch ecosystem to replicate state-of-the-art research.

The MUGEN playground provides examples for two primary multimodal tasks:

1. Text-video retrieval: Finding videos that match specific text descriptions.
2. Text-to-video generation: Generating video content based on text prompts.

Specific implementation details and scripts for these tasks are located in the retrieval and generation subdirectories respectively.

The MUGEN retrieval model is based on the VideoCLIP architecture, which is a contrastive model designed to learn a joint embedding space for video and text. It optimizes a scaled cosine similarity function between video and text embedding vectors.

Core Components:

• Video Encoder: Uses a Separable 3D CNN (S3D).
• Text Encoder: Uses DistilBERT (a lightweight transformer).

This architecture allows for retrieval tasks such as Text2video (finding videos given a text query) and Video2text (finding text descriptions given a video).

MUGEN follows a two-stage generative process inspired by DALL-E but adapted for video using VideoGPT components:

1. Discrete Latent Representation: Each modality is converted into discrete tokens.
   • Text: Obtained via tokenization (e.g., BPE).
   • Video/Image: A VQ-VAE model is used to learn downsampled discrete embedding vectors via nearest-neighbor lookups from a "codebook". The resulting indices are referred to as token IDs.
2. Joint Prior Learning: A GPT transformer decoder is used to learn a joint prior for both modalities in the latent space, allowing for generation in one modality given inputs from another.

For the video component specifically, the architecture uses 3D-convolutions and self-axial-attention within the VQ-VAE encoder/decoder and the GPT transformer decoder.

The repository is organized into several functional directories:

• torchmultimodal/diffusion_labs: Components for building diffusion models.
• torchmultimodal/models: Specific model architectures (e.g., blip2, albef).
• torchmultimodal/modules: Generic building blocks:
  • layers: Codebooks, patch embeddings, transformers.
  • losses: Contrastive loss, reconstruction loss.
  • encoders: ViT, BERT.
  • fusions: Fusion modules like Deep Set fusion.
• torchmultimodal/transforms: Model-specific data transforms (e.g., clip_transform, flava_transform, mae_transform).

Before running MDETR tasks, ensure you have followed the main TorchMultimodal installation instructions. Then, install the specific dependencies for MDETR using the provided requirements file.

pip install -r examples/mdetr/requirements.txt

To build from source and run the included examples, clone the repository recursively to ensure submodules are included, then install the package in editable mode.

git clone --recursive https://github.com/facebookresearch/multimodal.git multimodal
cd multimodal

pip install -e .

To run training and evaluation for Omnivore, you must install the nightly PyTorch core, the TorchMultimodal package, and specific dependencies like scipy and av.

1. Install nightly PyTorch core and domains using conda.
2. Install the TorchMultimodal package in editable mode from your local root folder.
3. Install additional dependencies scipy and av via pip.

# Install nightly pytorch core and domains
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch-nightly

# Install pytorch MultiModal
cd <torchmultimodal root folder>
pip install -e .

# Other dependency
pip install scipy==1.8.1 av==9.2.0

To evaluate the model, use the eval.py script. Configuration is managed via configs/eval.yaml.

By default, the evaluation script uses the MUGEN authors' published weights. To evaluate your own trained model, replace the checkpoint_path key in configs/eval.yaml with the path to your specific checkpoint file.

python eval.py config=configs/eval.yaml

Omnivore expects specific folder structures for ImageNet1K, Kinetics400, and SunRGBD datasets to support multi-modal training (image, video, and RGB-D).

ImageNet1K

Organize by class subdirectories under train and val folders:

root
    train
        class_1
            image_1.jpeg
            ...
    val
        class_1
            image_1.jpeg
            ...

Kinetics400

Organize by class subdirectories containing .mp4 files:

root
    train
        class_1
            video_1.mp4
            ...

SunRGBD

Requires a specific structure containing image directories, depth files, and metadata files (intrinsics.txt, scene.txt), along with the SUNRGBDtoolbox containing allsplit.mat.