repository·main·Indexed 26 days ago
https://github.com/bytedance/latentsyncAn end-to-end lip-sync method using audio-conditioned latent diffusion models and Stable Diffusion. The repository provides tools for inference via Gradio or CLI, a data processing pipeline, and training scripts for U-Net and SyncNet. It supports versions 1.5 and 1.6, with specific VRAM requirements and configurations for different training stages and resolutions (up to 512x512).
To install the required packages and download the necessary checkpoints (latentsync_unet.pt and whisper/tiny.pt), run the provided setup script. Alternatively, you can manually download the checkpoints from the HuggingFace repo.
source setup_env.shYou can perform lip-sync inference using a web-based Gradio interface. Note the minimum VRAM requirements:
python gradio_app.pyThe pipeline prepares video data by removing broken files, resampling FPS to 25 and audio to 16000 Hz, performing scene detection, splitting segments, applying affine transforms to faces (resized to 256x256), and filtering by sync confidence and hyperIQA scores.
Note: You must modify the input_dir parameter within the script to point to your data directory. Processed videos are saved in the high_visual_quality directory.
./data_processing_pipeline.shWhen using LatentSync 1.5, ensure your environment meets the following dependency requirements:
diffusers version must be 0.32.2 or higher.xFormers and Triton are no longer required dependencies.The current codebase is compatible with both LatentSync 1.5 and 1.6. To switch between these versions, you must perform two steps:
resolution parameter in the U-Net config file to match the version's training resolution (e.g., 512 $\times$ 512 for LatentSync 1.6).LatentSync 1.6 was trained on 512 $\times$ 512 resolution videos to resolve blurriness in teeth and lips observed in version 1.5.
Execute the inference script via the CLI. You can adjust the following parameters for better results:
inference_steps [20-50]: Higher values improve visual quality but increase generation time.guidance_scale [1.0-3.0]: Higher values improve lip-sync accuracy but may cause video distortion or jitter../inference.shBefore training, ensure data is processed via the pipeline and download the pretrained SyncNet checkpoint:
huggingface-cli download ByteDance/LatentSync-1.6 stable_syncnet.pt --local-dir checkpointsTo prepare the data files list, run:
python -m tools.write_fileslistThen, execute the training script. You must update the U-Net configuration files in configs/unet/ to specify your data directory, checkpoint save path, and other hyperparameters.
./train_unet.shIn LatentSync 1.5, the VRAM requirement for stage2 training has been reduced to 20 GB, making it possible to train on a single RTX 3090. This is achieved by training only the temporal layer and the audio cross-attention layer instead of full-parameter fine-tuning, and by using gradient checkpointing and PyTorch's native FlashAttention-2 implementation.
To use the efficient training configuration, use configs/unet/stage2_efficient.yaml.
If you need to train SyncNet on your own datasets, use the following script. The data processing requirements are identical to the U-Net pipeline. Loss charts will be saved in train_output_dir after validations_steps are completed.
./train_syncnet.shThe SyncNet audio encoder architecture is defined in a configuration file. It typically accepts a mel spectrogram as input and outputs a feature map.
Key configuration parameters:
in_channels: The number of input channels.block_out_channels: A list defining the number of channels at each layer. Deeper networks generally require larger numbers of channels.downsample_factors: Defines the downsampling at each layer. If the input resolution changes, you must redefine these factors so the output feature map maintains a D x 1 x 1 shape to allow for cosine similarity computation.attn_blocks: A list where 1 indicates the presence of a self-attention layer and 0 indicates its absence.dropout: The dropout rate.audio_encoder: # input (1, 80, 52)
in_channels: 1
block_out_channels: [32, 64, 128, 256, 512, 1024, 2048]
downsample_factors: [[2, 1], 2, 2, 1, 2, 2, [2, 3]]
attn_blocks: [0, 0, 0, 1, 1, 0, 0]
dropout: 0.0The SyncNet visual encoder architecture is defined in a configuration file.
Key configuration parameters:
in_channels: This must equal num_frames * image_channels.image_channels is 3.image_channels equals the latent_channels of the VAE being used (e.g., 4 for SD 1.5/SDXL, or 16 for FLUX/SD3).block_out_channels: A list defining the number of channels at each layer.downsample_factors: Defines the downsampling at each layer.attn_blocks: A list where 1 indicates the presence of a self-attention layer and 0 indicates its absence.dropout: The dropout rate.visual_encoder: # input (48, 128, 256)
in_channels: 48 # (16 x 3)
block_out_channels: [64, 128, 256, 256, 512, 1024, 2048, 2048]
downsample_factors: [[1, 2], 2, 2, 2, 2, 2, 2, 2]
attn_blocks: [0, 0, 0, 0, 1, 1, 0, 0]
dropout: 0.0Available configuration files in configs/unet/:
stage1.yaml: Stage 1 training (requires 23 GB VRAM).stage2.yaml: Optimal Stage 2 training (requires 30 GB VRAM).stage2_efficient.yaml: Efficient Stage 2 training for consumer GPUs like RTX 3090 (requires 20 GB VRAM; may have slight quality/consistency trade-offs).stage1_512.yaml: Stage 1 training at 512 $\times$ 512 resolution (requires 30 GB VRAM).stage2_512.yaml: Stage 2 training at 512 $\times$ 512 resolution (requires 55 GB VRAM).