DOCSEARCHbyAlgolia
Submit Docs

Wav2Lip

repository·master·Indexed 12 days ago

https://github.com/rudrabha/wav2lip

A tool for accurately lip-syncing videos to audio. It offers an open-source version for research and a commercial API via the Sync.so infrastructure with Python (syncsdk) and TypeScript (@sync.so/sdk) SDKs. Features include support for the lipsync-2 model, LSE-D and LSE-C quality metrics, and options for training standard or GAN-based models using datasets like LRS2, LRS3, and LRW.

Tokens
3.4K
Snippets
12
Records
20
Agent score
94%

What's inside Wav2Lip

  2. Guidelines for training on custom datasets

    master

    When training or fine-tuning on datasets other than LRS2, keep the following in mind:

    • Expert Discriminator: You MUST train a new expert discriminator for your specific dataset before training Wav2Lip.
    • Sync-Correction: If using web-scraped data, the videos often need to be sync-corrected.
    • FPS Consistency: The code is sensitive to the FPS of the videos; changing FPS requires significant code modifications.
    • Success Metrics: For good results, the expert discriminator's eval loss should reach approximately 0.25 and the Wav2Lip eval sync loss should reach approximately 0.2.

  4. Preprocess LRS2 dataset for training

    master

    Before training, you must preprocess the LRS2 dataset. The dataset should follow this structure:

    data_root (mvlrs_v1)
├── main
|   └── [five-digit numbered video IDs].mp4

    Place your LRS2 filelists (train.txt, val.txt, test.txt) in the filelists/ folder, then run:

    python preprocess.py --data_root data_root/main --preprocessed_root lrs2_preprocessed/
    python preprocess.py --data_root data_root/main --preprocessed_root lrs2_preprocessed/

  5. Download Wav2Lip model weights

    master

    You need to download the pre-trained weights to perform inference. There are two main versions available:

    • Wav2Lip: Highly accurate lip-sync.
    • Wav2Lip + GAN: Slightly lower lip-sync accuracy but offers better visual quality.

    Download links can be found in the 'Getting the weights' section of the repository documentation.

  6. Integrate Wav2Lip evaluation scripts with SyncNet

    master

    After setting up syncnet_python, you must copy the Wav2Lip evaluation scripts into the syncnet_python directory to run the calculations.

    1. Navigate to the Wav2Lip evaluation folder.
    2. Copy all .py and .sh files into the syncnet_python folder.
    cd Wav2Lip/evaluation/scores_LSE/
cp *.py syncnet_python/
cp *.sh syncnet_python/

  7. Access test filelists for LRS2, LRS3, and LRW

    master

    The evaluation/test_filelists/ directory contains filelists for evaluating the Wav2Lip model on standard datasets: LRS2, LRS3, and LRW. Each filelist contains the names of audio-video pairs from the respective test sets.

    Usage Restrictions:

    • The LRS2 and LRW filelists are strictly "Copyright BBC".
    • They are restricted to non-commercial research only.
    • Users must have an agreement with the BBC to access the Lip Reading in the Wild and/or Lip Reading Sentences in the Wild datasets.
    • For licensing details, visit: https://www.bbc.co.uk/rd/projects/lip-reading-datasets.

  8. Set up the evaluation environment for LSE-D and LSE-C metrics

    master

    To evaluate lip-sync quality using LSE-D and LSE-C metrics, you must set up a separate environment using the syncnet_python repository to avoid dependency conflicts with the main Wav2Lip installation.

    1. Clone the syncnet_python repository.
    2. Install dependencies and download pre-trained models within the cloned directory.
    3. Important: Use a separate virtual environment for these evaluation scripts to prevent version mismatches.
    git clone https://github.com/joonson/syncnet_python.git
cd syncnet_python
pip install -r requirements.txt
sh download_model.sh

  10. Make a lip sync generation using TypeScript

    master

    Use the SyncClient to submit a lip-sync generation job. The client.generations.create method accepts an array of input objects (type video or audio) and a model string (e.g., lipsync-2). After receiving a jobId, poll client.generations.get(jobId) until the status reaches COMPLETED or FAILED.

    import { SyncClient, SyncError } from "@sync.so/sdk";

const apiKey = "YOUR_API_KEY_HERE";
const videoUrl = "https://assets.sync.so/docs/example-video.mp4";
const audioUrl = "https://assets.sync.so/docs/example-audio.wav";

const client = new SyncClient({ apiKey });

async function main() {
    try {
        const response = await client.generations.create({
            input: [
                { type: "video", url: videoUrl },
                { type: "audio", url: audioUrl },
            ],
            model: "lipsync-2",
            options: { sync_mode: "cut_off" },
            outputFileName: "quickstart"
        });
        
        const jobId = response.id;
        let status = '';
        let generation;

        while (status !== 'COMPLETED' && status !== 'FAILED') {
            await new Promise(resolve => setTimeout(resolve, 10000));
            generation = await client.generations.get(jobId);
            status = generation.status;
        }

        if (status === 'COMPLETED') {
            console.log('output url:', generation?.outputUrl);
        }
    } catch (err) {
        if (err instanceof SyncError) {
            console.error(`Error: ${err.statusCode} ${JSON.stringify(err.body)}`);
        }
    }
}

main();