DOCSEARCHbyAlgolia
Submit Docs

SyncNet Python

repository·master·Indexed 21 days ago

https://github.com/joonson/syncnet_python

An audio-to-video synchronization network for removing temporal lags between audio and visual streams and identifying speakers in multi-face videos. Includes a face detector adapted from face-detection-pytorch and a three-stage pipeline for face detection/tracking, sync offset estimation, and visualization.

Tokens
685
Snippets
4
Records
5
Agent score
25%

What's inside syncnet_python

  2. Install SyncNet with GPU or CPU support

    master

    SyncNet can be installed using Conda environments. The code is designed to automatically detect and use a CUDA GPU if available, falling back to CPU if no GPU is detected.

    To install with GPU (CUDA) support:

    conda env create -f environment.yml

    To install for CPU only:

    conda env create -f environment-cpu.yml
    conda env create -f environment.yml

  4. Run the full SyncNet pipeline

    master

    The full synchronization pipeline consists of three distinct stages: face detection/tracking, sync offset estimation, and visualization. You must run these scripts sequentially.

    Arguments for all scripts:

    • --videofile: Path to the input video file.
    • --reference: A unique name for the video run (used for organizing output).
    • --data_dir: The base directory where outputs will be stored.

    Stage 1: Face Detection and Tracking

    python run_pipeline.py --videofile /path/to/video.mp4 --reference name_of_video --data_dir /path/to/output

    Stage 2: Sync Offset Estimation

    python run_syncnet.py --videofile /path/to/video.mp4 --reference name_of_video --data_dir /path/to/output

    Stage 3: Visualisation

    python run_visualise.py --videofile /path/to/video.mp4 --reference name_of_video --data_dir /path/to/output

    Output Locations

    Once completed, the results are stored in $DATA_DIR:

    • Cropped face tracks: $DATA_DIR/pycrop/$REFERENCE/*.avi
    • Output video: $DATA_DIR/pyavi/$REFERENCE/video_out.avi
    python run_pipeline.py --videofile /path/to/video.mp4 --reference name_of_video --data_dir /path/to/output
python run_syncnet.py --videofile /path/to/video.mp4 --reference name_of_video --data_dir /path/to/output
python run_visualise.py --videofile /path/to/video.mp4 --reference name_of_video --data_dir /path/to/output

  5. Run the SyncNet demo

    master

    To run a quick demonstration of the SyncNet synchronization network, use the demo_syncnet.py script. You must provide a video file and a temporary directory for processing.

    Arguments:

    • --videofile: Path to the input video file.
    • --tmp_dir: Path to a directory used for temporary files.

    Expected Output: The script should output values similar to these (exact values may vary by platform):

    • AV offset
    • Min dist
    • Confidence
    python demo_syncnet.py --videofile data/example.avi --tmp_dir /path/to/temp/directory