DOCSEARCHbyAlgolia
Submit Docs

Light-ASD: Lightweight Active Speaker Detection

repository·main·Indexed 41 days ago

https://github.com/junhua-liao/light-asd

A lightweight model for Active Speaker Detection (ASD) in video sequences. Includes pretrained weights and scripts for evaluation on the AVA-ActiveSpeaker and Columbia ASD datasets, as well as a demo utility for processing raw video files to identify active speakers.

Tokens
797
Snippets
3
Records
3
Agent score
2%

What's inside Light-ASD

  1. Run an Active Speaker Detection (ASD) demo

    main

    To run a demo on a raw video file (e.g., .mp4 or .avi):

    1. Place your video file in the demo folder (e.g., demo/0001.mp4).
    2. Run Columbia_test.py specifying the video name and folder.

    By default, the model uses weights trained on the AVA-ActiveSpeaker dataset. To use the TalkSet fine-tuned weights, provide the --pretrainModel argument.

    Output: The resulting video will be saved to demo/[videoName]/pyavi/video_out.avi. In the output, active speakers are marked with a green box and non-active speakers with a red box.

    # Standard demo (uses AVA weights)
python Columbia_test.py --videoName 0001 --videoFolder demo

# Demo using TalkSet fine-tuned weights
python Columbia_test.py --videoName 0001 --videoFolder demo --pretrainModel weight/finetuning_TalkSet.model

  2. Evaluate on the AVA-ActiveSpeaker dataset

    main

    To evaluate the model using the AVA-ActiveSpeaker dataset, you must first prepare the data, then you can either train the model from scratch or run evaluation using the provided pretrained weights.

    Data Preparation

    Download and preprocess the AVA dataset using the train.py script with the --download flag.

    Training

    Train the model on the AVA dataset using train.py. Training outputs include:

    • exps/exps1/score.txt: Output score file.
    • exps/exp1/model/model_00xx.model: Trained model checkpoints.
    • exps/exps1/val_res.csv: Predictions for the validation set.

    Testing

    To verify the pretrained model weights (which achieve mAP: 94.06% on the validation set), run the training script with the --evaluation flag.

    # Data preparation
python train.py --dataPathAVA AVADataPath --download 

# Training
python train.py --dataPathAVA AVADataPath

# Testing with pretrained weights
python train.py --dataPathAVA AVADataPath --evaluation

  3. Evaluate on the Columbia ASD dataset

    main

    You can evaluate the model on the Columbia ASD dataset using weights trained on AVA or weights fine-tuned on the TalkSet dataset.

    Using AVA-trained weights

    Run Columbia_test.py with the --evalCol flag. The dataset and labels will be downloaded to the specified --colSavePath.

    Using TalkSet fine-tuned weights

    For higher performance (Avg. F1: 95.5%), use the fine-tuned weights by specifying the --pretrainModel path.

    # Test with AVA weights
python Columbia_test.py --evalCol --colSavePath colDataPath

# Test with TalkSet fine-tuned weights
python Columbia_test.py --evalCol --pretrainModel weight/finetuning_TalkSet.model --colSavePath colDataPath