DOCSEARCHbyAlgolia
Submit Docs

Step-Video-T2V Documentation

repository·main·Indexed 25 days ago

https://github.com/stepfun-ai/step-video-t2v

A 30 billion parameter text-to-video model capable of generating up to 204 frames. It utilizes a high-compression Video-VAE and a DiT architecture with 3D full attention. The repository provides instructions for installation, multi-GPU parallel inference using torchrun, and recommended hyperparameters for Step-Video-T2V and Step-Video-T2V-Turbo.

Tokens
842
Snippets
2
Records
5
Agent score
35%

What's inside Step-Video-T2V

  1. Run Multi-GPU Parallel Inference

    main

    Step-Video-T2V uses a decoupling strategy for the text encoder, VAE decoding, and DiT to optimize GPU resource utilization. This requires a dedicated GPU to handle API services for text encoder embeddings and VAE decoding.

    1. Start the remote server to get the caption and VAE API URLs: python api/call_remote_server.py --model_dir <your_model_dir> &
    2. Run the parallel inference script using torchrun. Ensure that tp_degree (tensor parallel) multiplied by ulysses_degree equals the total parallel count.
    python api/call_remote_server.py --model_dir where_you_download_dir &  ## This returns the URL for caption and VAE APIs

parallel=4
url='127.0.0.1'
model_dir=where_you_download_dir

tp_degree=2
ulysses_degree=2

# make sure tp_degree x ulysses_degree = parallel
torchrun --nproc_per_node $parallel run_parallel.py --model_dir $model_dir --vae_url $url --caption_url $url  --ulysses_degree $ulysses_degree --tensor_parallel_degree $tp_degree --prompt "一名宇航员在月球上发现一块石碑，上面印有“stepfun”字样，闪闪发光" --infer_steps 50  --cfg_scale 9.0 --time_shift 13.0

  2. Install Step-Video-T2V

    main

    To install Step-Video-T2V, ensure you have Python >= 3.10, PyTorch >= 2.3-cu121, CUDA Toolkit, and FFmpeg installed. It is recommended to use Anaconda or Miniconda.

    Note: The self-attention in the text-encoder (step_llm) only supports CUDA capabilities sm_80, sm_86, and sm_90.

    git clone https://github.com/stepfun-ai/Step-Video-T2V.git
conda create -n stepvideo python=3.10
conda activate stepvideo

cd Step-Video-T2V
pip install -e .
pip install flash-attn --no-build-isolation  ## flash-attn is optional