Follow-Your-Pose

If you encounter stability issues with the standard xformers installation, you can attempt to install it using the following specific wheel:

wget https://github.com/ShivamShrirao/xformers-wheels/releases/download/4c06c79/xformers-0.0.15.dev0+4c06c79.d20221201-cp38-cp38-linux_x86_64.whl
pip install xformers-0.0.15.dev0+4c06c79.d20221201-cp38-cp38-linux_x86_64.whl

To fine-tune text-to-image diffusion models (like Stable Diffusion 1.4) for text-to-video generation, use the train_followyourpose.py script with accelerate. This command is configured for multi-GPU training on 8 GPUs.

TORCH_DISTRIBUTED_DEBUG=DETAIL accelerate launch \
    --multi_gpu --num_processes=8 --gpu_ids '0,1,2,3,4,5,6,7' \
    train_followyourpose.py \
    --config="configs/pose_train.yaml" 

Follow-Your-Pose is trained using CUDA 11, accelerate, and xformers on 8x A100 GPUs. To set up a local environment, create a new Conda environment and install the required dependencies.

It is highly recommended to use xformers on A100 GPUs to reduce memory usage and improve runtime. If standard installation is unstable, you can use a specific wheel.

The environment is similar to Tune-A-Video.

conda create -n fupose python=3.8
conda activate fupose

pip install -r requirements.txt

Once training is complete, run the txt2video.py script to generate videos. You must provide a configuration file and a path to a skeleton video.

Note on Skeleton Videos: You can generate skeleton videos using mmpose (using HRNet). Ensure you replace the background of your skeleton video with black.

TORCH_DISTRIBUTED_DEBUG=DETAIL accelerate launch \
    --gpu_ids '0' \
    txt2video.py \
    --config="configs/pose_sample.yaml" \
    --skeleton_path="./pose_example/vis_ikun_pose2.mov"

To perform inference, you need to download the pretrained checkpoints from Hugging Face and place them in the checkpoints directory following this structure:

FollowYourPose
├── checkpoints
│   ├── followyourpose_checkpoint-1000
│   │   ├──...
│   ├── stable-diffusion-v1-4
│   │   ├──...
│   └── pose_encoder.pth

The app.py file serves as the entrypoint for a Gradio-based web interface that allows users to perform pose-guided text-to-video generation. The interface accepts an input video (either raw or skeleton), a text prompt, and various sampling/cropping parameters to generate a new video.

To run the demo, ensure you have the necessary dependencies installed and execute the script. The application initializes a pipeline using merge_config_then_run() and launches a web server on 0.0.0.0:80.

Weights are hosted on HuggingFace. You must install Git LFS, clone the YueMafighting/FollowYourPose_v1 repository, and move the contents into a checkpoints/ directory within the project root.

%mkdir checkpoints
!git lfs install
!git clone https://huggingface.co/YueMafighting/FollowYourPose_v1
%mv /content/FollowYourPose/FollowYourPose_v1/* /content/FollowYourPose/checkpoints/
%rm -rf FollowYourPose_v1

You can run a local Gradio web interface for the model. This requires an A100 or 3090 GPU. After running the command, the demo will be available at a local URL (e.g., http://0.0.0.0:Port).

python app.py

If you use this project in your research, please cite the following paper:

@article{ma2023follow,
  title={Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos},
  author={Ma, Yue and He, Yingqing and Cun, Xiaodong and Wang, Xintao and Shan, Ying and Li, Xiu and Chen, Qifeng},
  journal={arXiv preprint arXiv:2304.01186},
  year={2023}
}

Use the txt2video.py script with accelerate launch to generate videos guided by a skeleton video.

Important Configuration Requirements:

1. GPU Memory Management: If you encounter memory issues, set video_length=8 in ./config/pose_sample.yaml.
2. Skeleton Alignment: Ensure the skeleton frame length (defined in ./followyourpose/pipelines/pipeline_followyourpose.py:422) matches the video_length specified in your config.

Generated videos are saved to ./output/samples.

%cd /content/FollowYourPose
!TORCH_DISTRIBUTED_DEBUG=DETAIL accelerate launch txt2video.py --config="configs/pose_sample.yaml" --skeleton_path="./pose_example/vis_ikun_pose2.mov"