SkyReels-V2 Documentation

SkyReels-V2 is an infinite-length film generative model utilizing a Diffusion Forcing framework. It is designed to overcome common limitations in video generation such as constrained duration, poor motion dynamics, and inadequate shot-aware generation.

Key capabilities include:

• Story Generation
• Image-to-Video Synthesis
• Camera Director functionality
• Multi-subject consistent video generation (via the Skyreels-A2 system)
• Elements-to-Video Generation

SkyReels-V2 is evaluated using two primary methods to compare its Text-to-Video (T2V) and Image-to-Video (I2V) capabilities against state-of-the-art models:

1. SkyReels-Bench (Human Evaluation): A custom benchmark with 1,020 text prompts assessing:

   • Instruction Adherence
   • Motion Quality
   • Consistency
   • Visual Quality

2. VBench (Automated Evaluation): An objective benchmark using longer version prompts to measure:

   • Total Score
   • Quality Score
   • Semantic Score

SkyReels-V2 demonstrates high performance in instruction adherence and consistency, outperforming open-source models like HunyuanVideo-13B and Wan2.1-14B in both human and automated assessments.

Use generate_video.py for standard T2V and I2V tasks.

For I2V:

• You must provide an input image using --image ${image_path}.
• Recommended --guidance_scale 5.0 and --shift 3.0 (Note: the table below suggests 5.0 for guidance and 5.0 for shift in some contexts, but the text explicitly recommends 5.0/3.0 for I2V).

VRAM Requirements:

• 540P (1.3B model): ~14.7GB
• 540P (14B model): ~43.4GB

# run Text-to-Video Generation
model_id=Skywork/SkyReels-V2-T2V-14B-540P
python3 generate_video.py \
  --model_id ${model_id} \
  --resolution 540P \
  --num_frames 97 \
  --guidance_scale 6.0 \
  --shift 8.0 \
  --fps 24 \
  --prompt "A serene lake..." \
  --offload \
  --teacache \
  --use_ret_steps \
  --teacache_thresh 0.3

Extend an existing video by passing the --video_path parameter to the generate_video_df.py script.

model_id=Skywork/SkyReels-V2-DF-14B-540P
# video extention
python3 generate_video_df.py \
  --model_id ${model_id} \
  --resolution 540P \
  --ar_step 0 \
  --base_num_frames 97 \
  --num_frames 120 \
  --overlap_history 17 \
  --prompt ${prompt} \
  --addnoise_condition 20 \
  --offload \
  --use_ret_steps \
  --teacache \
  --teacache_thresh 0.3 \
  --video_path ${video_path}

To install SkyReels-V2, clone the repository and install the required dependencies using pip. The recommended test environment uses Python 3.10.12.

# clone the repository.
git clone https://github.com/SkyworkAI/SkyReels-V2
cd SkyReels-V2
# Install dependencies. Test environment uses Python 3.10.12.
pip install -r requirements.txt

To address issues with large, deformable motions and violations of physical laws, SkyReels-V2 employs Reinforcement Learning (RL) using a specialized reward model and Direct Preference Optimization (DPO).

The Pipeline:

1. Data Collection: A semi-automatic pipeline combines automatically generated motion pairs with human annotations.
2. Reward Modeling: A specialized reward model is trained to capture generic motion quality differences between paired samples.
3. Optimization: The learned reward function guides sample selection for DPO to enhance motion quality without degrading text alignment or overall video quality.

To use the structural captioning and fusion features, you need to download both the SkyCaptioner-V1 model weights and the Qwen2.5-32B-Instruct model (used for caption fusion). Use the huggingface-cli to download them to your local directory.

# download SkyCaptioner-V1
huggingface-cli download Skywork/SkyCaptioner-V1 --local-dir /path/to/your_local_model_path

# download Qwen2.5-32B-Instruct
huggingface-cli download Qwen/Qwen2.5-32B-Instruct --local-dir /path/to/your_local_model_path2

SkyReels-V2 uses a Diffusion Forcing Transformer to enable infinite-length video generation.

How it works:

• Each token is assigned an independent noise level, allowing for arbitrary per-token denoising schedules.
• It functions as a partial masking mechanism: zero noise represents a fully unmasked token, while complete noise represents a fully masked token.
• The model uses cleaner tokens as conditional information to guide the recovery of noisy tokens.
• This allows the model to extend video generation indefinitely by conditioning on the last frames of the previous segment.

You can run interactive web interfaces for both structural captioning and caption fusion using Gradio.

# Launch Structural Captioning Demo
export SkyCaptioner_V1_Model_PATH="/path/to/your_local_model_path"
python scripts/gradio_struct_caption.py \
    --skycaptioner_model_path ${SkyCaptioner_V1_Model_PATH}

# Launch Caption Fusion Demo
export LLM_MODEL_PATH="/path/to/your_local_model_path2"
python scripts/gradio_fusion_caption.py \
    --fusioncaptioner_model_path ${LLM_MODEL_PATH}

To accelerate inference using multiple GPUs, use torchrun with the --use_usp flag. Example for 2 GPUs:

model_id=Skywork/SkyReels-V2-DF-14B-540P
# diffusion forcing synchronous inference
torchrun --nproc_per_node=2 generate_video_df.py \
  --model_id ${model_id} \
  --resolution 540P \
  --ar_step 0 \
  --base_num_frames 97 \
  --num_frames 257 \
  --overlap_history 17 \
  --prompt "A graceful white swan..." \
  --addnoise_condition 20 \
  --use_usp \
  --offload \
  --seed 42

Use --image ${image} to control the start frame and --end_image ${end_image} to control the end frame during generation.

model_id=Skywork/SkyReels-V2-DF-14B-540P
# start/end frame control
python3 generate_video_df.py \
  --model_id ${model_id} \
  --resolution 540P \
  --ar_step 0 \
  --base_num_frames 97 \
  --num_frames 97 \
  --overlap_history 17 \
  --prompt ${prompt} \
  --addnoise_condition 20 \
  --offload \
  --use_ret_steps \
  --teacache \
  --teacache_thresh 0.3 \
  --image ${image} \
  --end_image ${end_image}