DOCSEARCHbyAlgolia
Submit Docs

MotionLCM

repository·main·Indexed 19 days ago

https://github.com/dai-wenxun/motionlcm

A real-time controllable motion generation framework using Latent Consistency Models. MotionLCM enables high-efficiency human motion generation conditioned on text and spatial-temporal control signals, such as trajectories. The framework supports motion reconstruction, text-to-motion, and motion control via Motion ControlNet and Consistency Latent Tuning (CLT), with tools for generating and rendering SMPL meshes.

Tokens
2.1K
Snippets
7
Records
9
Agent score
16%

What's inside MotionLCM

  1. Download pretrained models and prepare datasets

    main

    1. Pretrained Models

    Download the pre-trained models for motion reconstruction, text-to-motion, and motion control using:

    bash prepare/download_pretrained_models.sh

    Models are stored in experiments_recons, experiments_t2m, and experiments_control.

    2. Datasets

    For text-to-motion tasks, you need the HumanML3D dataset. Follow the HumanML3D setup guide and then copy the processed dataset to the MotionLCM repository:

    cp -r ../HumanML3D/HumanML3D ./datasets/humanml3d

    Note: If you want to test quickly without a full dataset, you can download a tiny version:

    bash prepare/prepare_tiny_humanml3d.sh
    bash prepare/download_pretrained_models.sh

  2. Train MotionLCM and Motion ControlNet

    main

    Train MotionLCM

    Update configs/motionlcm_t2m.yaml and run:

    python -m train_motionlcm --cfg configs/motionlcm_t2m.yaml

    Train Motion ControlNet

    Update configs/motionlcm_control_s.yaml and run:

    python -m train_motion_control --cfg configs/motionlcm_control_s.yaml

    Joint Control Configuration: By default, ControlNet trains on the Pelvis joint. To train on all OmniControl joints (Pelvis, Left foot, Right foot, Head, Left wrist, and Right wrist), modify TRAIN_JOINTS in DATASET.HUMANML3D.CONTROL_ARGS within configs/motionlcm_control_s.yaml:

    TRAIN_JOINTS: [0, 10, 11, 15, 20, 21]

    Note that training on all joints results in a different checkpoint (e.g., ..._all.ckpt) compared to the pelvis-only training (..._pelvis.ckpt).

    python -m train_motionlcm --cfg configs/motionlcm_t2m.yaml
python -m train_motion_control --cfg configs/motionlcm_control_s.yaml

  3. Run MotionLCM demos (Reconstruction, Text-to-Motion, and Control)

    main

    MotionLCM provides three main functionalities via demo.py. Outputs are stored in ${cfg.TEST_FOLDER} / ${cfg.NAME} / demo_${timestamp}.

    1. Motion Reconstruction

    Reconstruct motions using ground truth (GT) motions from the HumanML3D test set:

    python demo.py --cfg configs/vae.yaml

    2. Text-to-Motion

    Using specific prompts and lengths from assets/example.txt:

    python demo.py --cfg configs/mld_t2m.yaml --example assets/example.txt
python demo.py --cfg configs/motionlcm_t2m.yaml --example assets/example.txt

    Using prompts from the HumanML3D test set:

    python demo.py --cfg configs/mld_t2m.yaml
python demo.py --cfg configs/motionlcm_t2m.yaml

    3. Motion Control

    Using Motion ControlNet:

    python demo.py --cfg configs/motionlcm_control_s.yaml

    Using Consistency Latent Tuning (CLT):

    python demo.py --cfg configs/motionlcm_t2m_clt.yaml --optimize
    python demo.py --cfg configs/motionlcm_t2m.yaml --example assets/example.txt

  4. Train Motion VAE and MLD models

    main

    To train the foundational Motion VAE and Motion Latent Diffusion (MLD) models, update configs/vae.yaml and configs/mld_t2m.yaml respectively, then execute the following commands:

    python -m train_vae --cfg configs/vae.yaml
python -m train_mld --cfg configs/mld_t2m.yaml

  5. Generate and render SMPL meshes

    main

    After running a demo, a .pkl file is generated containing joints, text, length, and optionally hint. To visualize these as SMPL meshes, follow these steps:

    1. Create SMPL meshes

    Use fit.py to convert the pickle file into a mesh numpy array (stored as *_mesh.pkl).

    For a single file:

    python fit.py --pkl assets/example.pkl

    For an entire directory:

    python fit.py --dir assets/

    2. Render SMPL meshes

    Requires TEMOS-Rendering motions Blender setup. Use the render.py script via Blender's command line.

    Mode: sequence (renders keyframes):

    YOUR_BLENDER_PATH/blender --background --python render.py -- --pkl assets/example_mesh.pkl --mode sequence --num 8

    Mode: video (renders an MP4):

    YOUR_BLENDER_PATH/blender --background --python render.py -- --pkl assets/example_mesh.pkl --mode video --fps 20

    Mode: frame (renders a specific frame):

    YOUR_BLENDER_PATH/blender --background --python render.py -- --pkl assets/example_mesh.pkl --mode frame --exact_frame 0.5
    python fit.py --pkl assets/example.pkl

  6. Install MotionLCM and set up the environment

    main

    To use MotionLCM, create a Conda environment with Python 3.10.12 and install the required dependencies from requirements.txt. The project is tested on Python 3.10.12 and PyTorch 1.13.1.

    1. Conda Environment

    conda create python=3.10.12 --name motionlcm
conda activate motionlcm
pip install -r requirements.txt

    2. System Dependencies

    Install ffmpeg for stick figure visualization and git-lfs for large file handling.

    Using sudo (if permitted):

    sudo apt update
sudo apt install ffmpeg

    Using Conda (if no sudo access):

    conda install conda-forge::ffmpeg
conda install conda-forge::git-lfs

    3. Download Materials

    Run the following scripts to download necessary dependencies (GloVe, T5, SMPL models, etc.):

    bash prepare/download_glove.sh
bash prepare/download_t2m_evaluators.sh
bash prepare/prepare_t5.sh
bash prepare/download_smpl_models.sh
    #!/bin/bash
# Example of the required setup commands
conda create python=3.10.12 --name motionlcm
conda activate motionlcm
pip install -r requirements.txt

  7. Evaluate MotionLCM models

    main

    Use the test module to evaluate trained models.

    Motion Reconstruction

    python -m test --cfg configs/vae.yaml

    Text-to-Motion

    python -m test --cfg configs/mld_t2m.yaml
python -m test --cfg configs/motionlcm_t2m.yaml

    To adjust inference steps, modify num_inference_steps in:

    • configs/modules/scheduler_ddim.yaml (for MLD)
    • configs/modules/scheduler_lcm.yaml (for MotionLCM)

    Motion Control

    MotionLCM with ControlNet:

    python -m test --cfg configs/motionlcm_control_s.yaml

    MotionLCM with Consistency Latent Tuning (CLT):

    python -m test --cfg configs/motionlcm_t2m_clt.yaml --optimize

    Note: For CLT, num_inference_steps must remain 1 and batch_size must remain 1. Do not modify these.

    # Motion Reconstruction
python -m test --cfg configs/vae.yaml

# Text-to-Motion
python -m test --cfg configs/mld_t2m.yaml
python -m test --cfg configs/motionlcm_t2m.yaml

# Motion Control (ControlNet)
python -m test --cfg configs/motionlcm_control_s.yaml

# Motion Control (CLT)
python -m test --cfg configs/motionlcm_t2m_clt.yaml --optimize

  8. Configure training and testing parameters in YAML

    main

    Model training and testing parameters are managed via YAML configuration files (e.g., configs/motionlcm_t2m.yaml). Key parameters include:

    • ${FOLDER}: The base directory for the specific task (e.g., experiments_recons, experiments_t2m, or experiments_control).
    • ${TEST_FOLDER}: The base directory for testing tasks (e.g., experiments_recons_test, experiments_t2m_test, or experiments_control_test).
    • ${NAME}: The model name (e.g., motionlcm_humanml). The final output path is constructed as ${FOLDER}/${NAME}/${TIMESTAMP}.
    • ${TRAIN.PRETRAINED}: Path to the pre-trained model used for training.
    • ${TEST.CHECKPOINTS}: Path to the checkpoint used for testing.

  9. Configure Motion Control testing joints and density

    main

    When evaluating Motion Control models, you can configure which joints are used for testing and the density of control points.

    Testing Joints: In configs/motionlcm_control_s.yaml, set TEST_JOINTS. If you trained on all joints, you can choose from [0] (Pelvis), [10], [11], [15], [20], or [21].

    Testing Density: Set TEST_DENSITY to control the number of points sampled from the ground truth trajectory:

    • Percentage mode: Use 100 or 25 (e.g., 100% of points).
    • Absolute mode: Use 5, 2, or 1 (e.g., exactly 5 points).

    Logic implementation reference:

    # MotionLCM/mld/data/humanml/dataset.py
length = joints.shape[0]
density = self.testing_density
if density in [1, 2, 5]:
    choose_seq_num = density
else:
    choose_seq_num = int(length * density / 100)