SVFR Documentation

SVFR requires both Stable Video Diffusion and specific SVFR checkpoints.

1. Stable Video Diffusion: Use git-lfs to clone the stable-video-diffusion-img2vid-xt model into the models/ directory.
2. SVFR Checkpoints: Download the SVFR checkpoints manually from Google Drive and organize them in the models/ directory as shown below.

# Download Stable Video Diffusion
conda install git-lfs
git lfs install
git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt models/stable-video-diffusion-img2vid-xt

└── models
    ├── face_align
    │   ├── yoloface_v5m.pt
    ├── face_restoration
    │   ├── unet.pth
    │   ├── id_linear.pth
    │   ├── insightface_glint360k.pth
    └── stable-video-diffusion-img2vid-xt
        ├── vae
        ├── scheduler
        └── ...

The LQ2VideoLongSVDPipeline is a diffusion pipeline designed to generate video from an input image using Stable Video Diffusion (SVD). It supports various conditioning inputs including reference images, concatenated reference images, and ID prompts.

Key features include:

• Classifier-Free Guidance (CFG): Controlled via min_guidance_scale and max_guidance_scale.
• Noise Augmentation: Use noise_aug_strength to control how much the output deviates from the input image (higher values increase motion).
• Memory Management: Use decode_chunk_size to decode frames in smaller batches to prevent Out-of-Memory (OOM) errors.
• Deterministic Generation: Pass a torch.Generator to the generator argument.

Returns a LQ2VideoSVDPipelineOutput object containing frames and latents.

from src.pipelines.pipeline import LQ2VideoLongSVDPipeline

# Assuming components (vae, image_encoder, unet, scheduler, feature_extractor) are initialized
pipeline = LQ2VideoLongSVDPipeline(
    vae=vae, 
    image_encoder=image_encoder, 
    unet=unet, 
    scheduler=scheduler, 
    feature_extractor=feature_extractor
)

# Generate video
output = pipeline(
    ref_image=my_pil_image,
    ref_concat_image=None,  # Optional additional reference
    id_prompts=id_embeddings_tensor,
    task_id_input=task_tensor,
    height=512,
    width=512,
    num_frames=25,
    num_inference_steps=25,
    min_guidance_scale=1.0,
    max_guidance_scale=3.0,
    noise_aug_strength=0.02,
    decode_chunk_size=8,
    output_type="pil"
)

frames = output.frames

The forward method performs the spatio-temporal denoising pass. It accepts noisy video frames, timesteps, and conditioning information.

Arguments

• sample (torch.FloatTensor): Noisy input tensor of shape (batch, num_frames, channel, height, width).
• timestep (Union[torch.Tensor, float, int]): The current timestep for denoising.
• encoder_hidden_states (torch.Tensor): Conditioning states of shape (batch, sequence_length, cross_attention_dim).
• added_time_ids (torch.Tensor): Additional time IDs of shape (batch, num_additional_ids) used for sinusoidal embeddings.
• pose_cond_fea (Optional[torch.Tensor]): Optional pose conditioning features to be added to the sample.
• return_dict (bool, default: True): If True, returns a UNet3DConditionSVDOutput object; otherwise, returns a tuple containing the sample.

Returns

• UNet3DConditionSVDOutput (if return_dict=True) containing sample: A tensor of shape (batch, num_frames, out_channels, height, width).

# Assuming model is an instance of UNet3DConditionSVDModel
output = model(
    sample=noisy_video_tensor,  # (B, F, C, H, W)
    timestep=500,
    encoder_hidden_states=context_embeddings,
    added_time_ids=time_ids,
    pose_cond_fea=pose_features,  # Optional
    return_dict=True
)

restored_sample = output.sample

The dataset utilities provide functions to manipulate bounding boxes (bboxes) to ensure they are properly centered and shaped for model input.

Get Union Bounding Box

get_union_bbox(bboxes) takes a list of bounding boxes and returns a single bounding box that encompasses all of them.

• Input format: A list of [min_x, min_y, max_x, max_y].
• Output format: A numpy array [min_x, min_y, max_x, max_y].

Expand and Square Bounding Box

process_bbox(bbox, expand_radio, height, width) is a high-level utility that performs two steps:

1. Expansion: Expands the original bbox by a given expand_radio (ratio) relative to its width and height, clipped to the image boundaries.
2. Squaring: Converts the expanded bounding box into a square box centered around the original face area.

Parameters:

• bbox: The original bounding box [x1, y1, x2, y2].
• expand_radio: The multiplier used to expand the box dimensions.
• height: The height of the source image.
• width: The width of the source image.

Returns:

• A list representing the processed square bounding box [x1, y1, x2, y2].

# Example usage
# bbox format: [x1, y1, x2, y2]
processed_bbox = process_bbox(original_bbox, expand_radio=0.2, height=512, width=512)

The AlignImage class provides a high-level interface for detecting faces and extracting facial landmarks (5 keypoints) from an image using a YOLO-based detector. It can be initialized with a specific device and a checkpoint path, and it can be called directly on an image to retrieve keypoints, confidence scores, and bounding boxes.

When calling the instance, you can use the maxface parameter. If maxface=True and multiple faces are detected, the class will only return the landmarks, scores, and bounding boxes for the face with the largest area.

from src.dataset.face_align.align import AlignImage
import torch

# Initialize the aligner
aligner = AlignImage(device='cuda', det_path='checkpoints/yoloface_v5m.pt')

# Process an image (im is a torch tensor or compatible image representation)
# maxface=True ensures only the largest face is returned if multiple are found
five_pts_list, scores_list, bboxes_list = aligner(im, maxface=True)