DeepfakeBench

DeepfakeBench is a comprehensive, unified benchmark platform for deepfake detection (NeurIPS 2023 D&B). It provides a standardized framework for implementing, training, and evaluating state-of-the-art (SoTA) detection methods for both images and videos.

Key capabilities include:

• Unified Platform: Standardizes deepfake detection benchmarking.
• Data Management: A unified system ensuring consistent input across different models.
• Integrated Framework: Supports training, data loading, and evaluation at both image and video levels.
• Standardized Evaluations: Implements consistent metrics including frame-level AUC, video-level AUC, ACC (fake and real), EER, PR, and AP.
• Advanced Features: Supports multi-GPU training via DDP (Distributed Data Parallel) and high-speed IO using LMDB.

DeepfakeBench provides a comprehensive collection of 36 detectors categorized by their approach to deepfake detection. These include:

• Naive Detectors (5): Xception, MesoNet, MesoInception, CNN-Aug, and EfficientNet-B4.
• Spatial Detectors (20): Includes methods like Capsule, Face X-ray, FFD, CORE, RECCE, CLIP, SBI, and more.
• Frequency Detectors (3): F3Net, SPSL, and SRM.
• Video Detectors (8): TALL, I3D, STIL, FTCN, X-CLIP, TimeTransformer, and VideoMAE.

Each detector is implemented as a Python module within the training/detectors/ directory.

DeepfakeBench (v2) supports a wide range of detection methods and technical features:

Supported Detectors

Currently supports 36 detection methods:

• 28 image detectors
• 8 video detectors

Recent additions include SoTA detectors such as:

• Effort (ICML'25 spotlight)
• LSDA (CVPR'24)
• AltFreezing (CVPR'23)
• TALL (ICCV'23)
• IID (CVPR'23)
• SBI (CVPR'22)
• SLADD (CVPR'22)
• FTCN (ICCV'21)

Technical Capabilities

• Multi-GPU Training: Uses DDP for distributed training.
• Fast IO: Provides LMDB support for efficient data loading.
• Comprehensive Metrics: Evaluates performance using frame-level AUC, video-level AUC, ACC (fake and real), EER, PR, and AP.

Preprocessing involves face detection, alignment, and cropping.

Note: If you use the pre-processed data provided by the authors, you can skip preprocess.py, but you must still run rearrange.py to generate the required JSON files.

Steps to preprocess:

1. Download shape_predictor_81_face_landmarks.dat and place it in ./preprocessing/dlib_tools.
2. Edit ./preprocessing/config.yaml:
   • Set default to your target dataset name (e.g., FaceForensics++).
   • Set dataset_root_path to your actual dataset directory (default is ./datasets).
3. Run the preprocessing script:

cd preprocessing
python preprocess.py

You can set up the environment using one of two methods:

Option 1: Conda (Manual Setup)

Clone the repository and use a Conda environment with Python 3.7.2. Run the provided install.sh script to complete the installation.

Option 2: Docker (Recommended)

Use the provided Dockerfile to ensure environment consistency. This method is recommended to avoid environment-related issues. Note that Docker version 19.03.14 was used for the original setup, though later versions may be compatible.

# Option 1: Conda
git clone git@github.com:SCLBD/DeepfakeBench.git
cd DeepfakeBench
conda create -n DeepfakeBench python=3.7.2
conda activate DeepfakeBench
sh install.sh

# Option 2: Docker
docker build -t DeepfakeBench .
docker run --gpus all -itd -v /path/to/this/repository:/app/ --shm-size 64G DeepfakeBench

This module provides an unofficial PyTorch re-implementation of the Face X-ray paper's BI (Blending Image) data generation pipeline. You can generate blended images by running the bi_online_generation.py script, which produces results consistent with Figure 5 in the original paper.

To generate a complete BI dataset, you must first ensure all faces are cropped and landmarks are computed according to the logic defined in the codebase.

python bi_online_generation.py

To enable unified data loading across different datasets, you must run the rearrangement script. This generates JSON files in ./preprocessing/dataset_json that organize videos by labels and data splits (train, test, validation).

This step is required whether you are preprocessing your own raw data or using pre-processed data provided by the authors.

cd preprocessing
python rearrange.py

To perform cross-dataset evaluation, use training/test.py.

Requirement: You must place pre-trained weights in the ./training/weights folder before running evaluation.

python3 training/test.py \
--detector_path ./training/config/detector/xception.yaml \
--test_dataset "Celeb-DF-v1" "Celeb-DF-v2" "DFDCP" \
--weights_path ./training/weights/xception_best.pth