SteadyDancer

Before generating animation, you must extract and align poses from a reference image and a driving video. This process creates both 'positive' and 'negative' condition folders required by the model.

1. Positive Condition: Use preprocess/pose_align.py to extract aligned poses.
2. Negative Condition: Use preprocess/pose_align_withdiffaug.py to extract poses with different augmentations.
3. Image/Video Dumping: Use preprocess/dump_video_images.py to convert the resulting pose videos into image sequences.
4. File Preparation: Ensure the ref_image.png, driving_video.mp4, and prompt.txt are copied into the same output directory as the pose folders.

## Extract and align pose (Positive Condition)

outfn=$output/positive/all.mp4
outfn_align_pose_video=$output/positive/single.mp4
python preprocess/pose_align.py \
    --imgfn_refer "$ref_image_path" \
    --vidfn "${driving_video_path}/video.mp4" \
    --outfn "$outfn" \
    --outfn_align_pose_video "$outfn_align_pose_video"

python preprocess/dump_video_images.py "$outfn_align_pose_video" "$(dirname "$outfn_align_pose_video")"

## Extract and align pose (Negative Condition)

outfn=$output/negative/all.mp4
outfn_align_pose_video=$output/negative/single.mp4
python preprocess/pose_align_withdiffaug.py \
    --imgfn_refer "$ref_image_path" \
    --vidfn "${driving_video_path}/video.mp4" \
    --outfn "$outfn" \
    --outfn_align_pose_video "$outfn_align_pose_video"

python preprocess/dump_video_images.py "$outfn_align_pose_video" "$(dirname "$outfn_align_pose_video")"

Follow these steps to set up the environment. This assumes a Linux environment with CUDA support (e.g., CUDA 12.1).

1. Clone the repository
2. Create and activate a Conda environment
3. Install animation generation dependencies (including PyTorch, Flash Attention, and xformers)
4. Install pose extraction dependencies (including moviepy, decord, and the OpenMMLab stack: mmengine, mmcv, mmdet, and mmpose).

# Clone this repository
git clone https://github.com/MCG-NJU/SteadyDancer.git
cd SteadyDancer

# Create and activate conda environment
conda create -n steadydancer python=3.10 -y
conda activate steadydancer

# Install animate generation dependencies (Pytorch 2.5.1, CUDA 12.1 for example)
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.5cxx11abiFALSE-cp310-cp310-linux_x86_64.whl && python -c "import flash_attn"
pip install xformers==0.0.29.post1
pip install "xfuser[diffusers,flash-attn]"
pip install -r requirements.txt

# Install pose extraction dependencies
pip install --no-cache-dir -U pip setuptools wheel
pip install moviepy decord              # moviepy-2.2.1, decord-0.6.0
pip install --no-cache-dir -U openmim   # openmim-0.3.9
mim install mmengine                    # mmengine-0.10.7
mim install "mmcv==2.1.0"               # mmcv-2.1.0
mim install "mmdet>=3.1.0"              # mmdet-3.3.0
pip install mmpose                      # mmpose-1.3.2

Use generate_dancer.py to create the final animation. You can run this on a single GPU or across multiple GPUs using FSDP and xDiT USP.

Note on Reproducibility: Multi-GPU inference may produce different results than Single-GPU inference due to the non-deterministic nature of distributed computing. For better reproducibility, Single-GPU inference is recommended.

Required Input Directory Structure

The input_dir must contain:

• ref_image.png
• driving_video.mp4
• prompt.txt
• positive/ (folder containing aligned pose images)
• negative/ (folder containing augmented pose images)

# Single-GPU inference
CUDA_VISIBLE_DEVICES=0 python generate_dancer.py \
    --task i2v-14B --size 1024*576 \
    --ckpt_dir $ckpt_dir \
    --prompt "$prompt" \
    --image $image \
    --cond_pos_folder $cond_pos_folder \
    --cond_neg_folder $cond_neg_folder \
    --sample_guide_scale $cfg_scale \
    --condition_guide_scale $condition_guide_scale \
    --end_cond_cfg $pro \
    --base_seed $base_seed \
    --save_file "${save_file}--$(date +"%Y%m%d%H%M%S")"

# Multi-GPU inference using FSDP + xDiT USP
GPUs=2
torchrun --nproc_per_node=${GPUs} generate_dancer.py \
    --dit_fsdp --t5_fsdp --ulysses_size ${GPUs} \
    --task i2v-14B --size 1024*576 \
    --ckpt_dir $ckpt_dir \
    --prompt "$prompt" \
    --image $image \
    --cond_pos_folder $cond_pos_folder \
    --cond_neg_folder $cond_neg_folder \
    --sample_guide_scale $cfg_scale \
    --condition_guide_scale $condition_guide_scale \
    --end_cond_cfg $pro \
    --base_seed $base_seed \
    --save_file "${save_file}--$(date +"%Y%m%d%H%M%S")--xDiTUSP${GPUs}"

To run the model, you need to download the DW-Pose pretrained weights and the SteadyDancer-14B model weights.

DW-Pose weights:

• Download dw-ll_ucoco_384.pth using huggingface-cli into ./preprocess/pretrained_weights/dwpose.
• Download yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth using wget into the same directory.

SteadyDancer-14B weights:

• Use huggingface-cli or modelscope to download the weights into the ./SteadyDancer-14B directory.

# Download DW-Pose pretrained weights
mkdir -p ./preprocess/pretrained_weights/dwpose
huggingface-cli download yzd-v/DWPose --local-dir ./preprocess/pretrained_weights/dwpose --include "dw-ll_ucoco_384.pth"
wget https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_l_8x8_300e_coco/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth -O ./preprocess/pretrained_weights/dwpose/yolox_l_8x8_300e_coco.pth

# Download SteadyDancer-14B model weights
huggingface-cli download MCG-NJU/SteadyDancer-14B --local-dir ./SteadyDancer-14B
# or download from modelscope
# modelscope download --model MCG-NJU/SteadyDancer-14B ./SteadyDancer-14B

If you encounter ModuleNotFoundError: No module named 'mmcv._ext' during installation, you must manually build mmcv with CUDA operations.

Steps:

1. Uninstall existing mmcv, mmcv-full, mmcv-lite, mmpose, mmdet, and mmengine.
2. Re-install mmengine via mim.
3. Clone the mmcv repository and checkout version v2.1.0.
4. Build and install mmcv with MMCV_WITH_OPS=1.
5. Re-install mmdet and mmpose.
6. Verify the installation using the provided smoke test script.

# Clean and prep
pip uninstall -y mmcv mmcv-full mmcv-lite mmpose mmdet mmengine || true
mim install mmengine                    # mmengine-0.10.7

# Build mmcv with CUDA ops
git clone https://github.com/open-mmlab/mmcv.git
cd mmcv && git checkout v2.1.0
pip install -r requirements/optional.txt
gcc --version                                                   # Check the gcc version (requires 5.4+)
MMCV_WITH_OPS=1 MAX_JOBS=$(nproc) python setup.py build_ext     # Build the C++ and CUDA extensions
MMCV_WITH_OPS=1 MAX_JOBS=$(nproc) python setup.py develop       # Install mmcv with the C++ and CUDA extensions
python .dev_scripts/check_installation.py                       # Verify the mmcv installation
cd ../

# Reinstall deps that rely on mmcv
mim install "mmdet>=3.1.0"              # mmdet-3.3.0
pip install mmpose                      # mmpose-1.3.2

# Quick smoke test
python - <<'PY'
import mmcv, mmpose
from mmpose.apis import inference_topdown, init_model
from mmpose.evaluation.functional import nms
from mmpose.utils import adapt_mmdet_pipeline
from mmpose.structures import merge_data_samples
print("mmcv", mmcv.__version__, "mmpose", mmpose.__version__)
PY