LeviTor: 3D Trajectory-Oriented Image-to-Video Synthesis

LeviTor requires several external and project-specific checkpoints. Create a checkpoints directory and organize them as follows:

1. Depth Anything V2: Download depth_anything_v2_vitl.pth from Depth Anything V2.
2. Segment Anything (SAM): Download sam_vit_h_4b8939.pth from Segment Anything.
3. Stable Video Diffusion (SVD): Download stable-video-diffusion-img2vid-xt from stabilityai.
4. LeviTor Checkpoints: Create a LeviTor subdirectory inside checkpoints and download the weights from LeviTor Hugging Face.

Required Directory Structure:

checkpoints/
 |-- sam_vit_h_4b8939.pth
 |-- depth_anything_v2_vitl.pth
 |-- stable-video-diffusion-img2vid-xt/
 |-- LeviTor/
     |-- random_states_0.pkl
     |-- scaler.pt
     |-- scheduler.bin
     |-- controlnet/
     |-- unet/

The Gradio interface allows for 3D trajectory-oriented video synthesis through the following workflow:

1. Upload Start Image: Use the Upload Start Image button to provide your source image.
2. Select Area (SAM):
   • Click Select Area with SAM and click on the image.
   • Use the Add SAM Point? box: Input 1 if the clicked point is inside the target area, or 0 if it is outside (to define the background/uninterested area).
3. Draw 3D Trajectory:
   • Click Add New Drag Trajectory and click a series of points to draw a 2D path.
   • Observe the Depths for reference box to see the depth values of your clicked points.
   • In the Input depth values here box, enter normalized depth values (range 0 to 1).
     • Note: These are relative depths. Smaller values move the area closer to the camera; larger values move it further away.
     • The number of input values must match the number of points in the reference box.
4. Configure Inference:
   • In Input number of points for inference here, enter a ratio.
     • Small ratio: Better for non-rigid body motion.
     • Large ratio: Better for rigid body motion.
5. Generate: Click Run to synthesize the video.

Pro-tips:

• Multiple Trajectories: You can draw multiple trajectories to control different parts of the scene.
• Orbiting Effects: To create an orbiting effect, select a reference object and create a stationary path for it. Set its depth (e.g., 0.1). Then, adjust the orbiting object's depth values relative to that: values > 0.1 move it behind the reference, and values < 0.1 move it in front.

The easiest way to use CoTracker is to load a pretrained model from torch.hub. In offline mode, you pass the entire video tensor to the model at once. This is suitable for videos where the full content is available.

Requirements:

• pip install imageio[ffmpeg]
• A video loaded as a tensor with shape [B, T, C, H, W] (Batch, Time, Channels, Height, Width).

import torch
import imageio.v3 as iio

# Load video
url = 'https://github.com/facebookresearch/co-tracker/blob/main/assets/apple.mp4'
frames = iio.imread(url, plugin="FFMPEG")

device = 'cuda'
grid_size = 10
# Prepare video tensor: B T C H W
video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device)

# Load and run CoTracker2
cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2").to(device)
pred_tracks, pred_visibility = cotracker(video, grid_size=grid_size)
# pred_tracks shape: [B, T, N, 2]
# pred_visibility shape: [B, T, N, 1]

Execute the interactive Gradio demo using the following command. Ensure you are in the LeviTor root directory and have the checkpoints folder populated.

Command Arguments:

• --frame_interval: Interval between frames.
• --num_frames: Total number of frames to generate.
• --pretrained_model_name_or_path: Path to the SVD checkpoint.
• --resume_from_checkpoint: Path to the LeviTor checkpoint directory.
• --width / --height: Output resolution.
• --seed: Random seed for reproducibility.
• --mixed_precision: Precision mode (e.g., fp16).
• --enable_xformers_memory_efficient_attention: Enables memory-efficient attention via xformers.
• --output_dir: Directory to save generated videos.
• --gaussian_r: Gaussian radius parameter.
• --sam_path: Path to the SAM checkpoint.
• --depthanything_path: Path to the Depth Anything V2 checkpoint.

python gradio_demo/gradio_run.py --frame_interval 1 --num_frames 16 --pretrained_model_name_or_path checkpoints/stable-video-diffusion-img2vid-xt --resume_from_checkpoint checkpoints/LeviTor --width 288 --height 512 --seed 217113 --mixed_precision fp16 --enable_xformers_memory_efficient_attention --output_dir ./outputs --gaussian_r 10 --sam_path checkpoints/sam_vit_h_4b8939.pth --depthanything_path checkpoints/depth_anything_v2_vitl.pth

You can integrate the metric depth model into your Python workflows using the DepthAnythingV2 class.

Key Parameters:

• encoder: One of 'vits', 'vitb', or 'vitl'.
• max_depth: Set to 20 for indoor (Hypersim) models and 80 for outdoor (Virtual KITTI 2) models.

Method:

• infer_image(raw_img): Takes a numpy image and returns an $H imes W$ depth map in meters as a numpy array.

import cv2
import torch
from depth_anything_v2.dpt import DepthAnythingV2

model_configs = {
    'vits': {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]},
    'vitb': {'encoder': 'vitb', 'features': 128, 'out_channels': [96, 192, 384, 768]},
    'vitl': {'encoder': 'vitl', 'features': 256, 'out_channels': [256, 512, 1024, 1024]}
}

encoder = 'vitl' # or 'vits', 'vitb'
dataset = 'hypersim' # 'hypersim' for indoor model, 'vkitti' for outdoor model
max_depth = 20 # 20 for indoor model, 80 for outdoor model

model = DepthAnythingV2(**{**model_configs[encoder], 'max_depth': max_depth})
model.load_state_dict(torch.load(f'checkpoints/depth_anything_v2_metric_{dataset}_{encoder}.pth', map_location='cpu'))
model.eval()

raw_img = cv2.imread('your/image/path')
depth = model.infer_image(raw_img) # HxW depth map in meters in numpy