clean-fid

The Clean-FID leaderboard compares three types of scores:

1. Legacy-FID (reported): The original FID score reported in the model's source paper.
2. Legacy-FID (reproduced): The FID score obtained by reproducing the original paper's methodology using the project's tools.
3. Clean-FID: The score computed using the Clean-FID method proposed in this repository.

Note that for some models (marked with †), the reference distribution used was the training set compared against 50k generated images, rather than the standard test set comparison.

To prepare the LSUN cat dataset:

1. Download the lmdb files from here.
2. Use dataset_tool.py to center crop and resize 200k images into a zip file.
3. Unzip the file into a folder (e.g., lsuncat200k/).
4. Use scripts/process_dataset_folder.py to pick 50,000 images and generate statistics using --mode legacy_tensorflow.

# Step 1: Convert LMDB to cropped images
python dataset_tool.py --source=~/datasets/lsun/lmdb/cat \
    --dest= lsuncat200k.zip --transform=center-crop \
    --width=256 --height=256 --max_images=200000

# Step 2: Generate statistics from the unzipped folder
python scripts/process_dataset_folder.py \
    --input_folder ~/cleanfid_images/LSUN_CAT/lsuncat200k \
    --num_images 50000 --seed 0 --mode legacy_tensorflow \
    --output_file stats/lsuncat_legacy_tensorflow_train_256.npz

Execute the test suite using the provided bash script. After running, inspect the logfile to ensure no errors were thrown.

bash tests/tests_main.sh

To prepare Cifar-10:

1. Download the dataset from the original authors.
2. Convert the 50k training images to a folder of images using dataset_tool.py from the stylegan2-ada-pytorch repository.
3. Run scripts/process_dataset_folder.py on the train and test image folders to generate .npz statistics files for both legacy_tensorflow and clean modes.

# Example: Generate statistics for Cifar-10 training split
python scripts/process_dataset_folder.py \
    --input_folder ~/datasets/cifar10_train_images \
    --mode clean \
    --output_file stats/cifar10_clean_train_32.npz

If you need to replicate results from the original pytorch-fid implementation, use the mode="legacy_pytorch" flag in the fid.compare_folders function.

Note that CleanFID differs from the legacy implementation in its use of weights (which are ported from TensorFlow) and its use of the bilinear filter via torch.nn.functional.interpolate for resizing. Using the legacy_pytorch mode ensures compatibility with the popular PyTorch implementation.

from cleanfid import fid

score = fid.compare_folders("test_fake/", "test_real/", mode="legacy_pytorch")

To compile the project into a distributable wheel format locally, use the setup.py script with the bdist_wheel command.

python setup.py bdist_wheel

To verify that clean-fid with mode="legacy_tensorflow" produces the same results as the official TTUR implementation, follow these steps:

1. Compute score with CleanFID:

   from cleanfid import fid
   score = fid.compare_folders("test_fake/", "test_real/", mode="legacy_tensorflow")

2. Compute score with official TTUR: Install the required legacy dependencies and run the original script:

   pip install tensorflow-gpu==1.14
   pip install imageio
   cd test && git clone https://github.com/bioinf-jku/TTUR
   python fid.py test_fake/ test_real/

Comparing these two outputs allows you to confirm the implementation parity.

# CleanFID verification steps
from cleanfid import fid
score = fid.compare_folders("test_fake/", "test_real/", mode="legacy_tensorflow")

If you need to build the package from the source code, use the following commands to create a wheel and install it via pip:

python setup.py bdist_wheel
pip install dist/*

To use the library for evaluating generative models with consistent FID/KID scores, install it using pip:

pip install clean-fid

If you need to replicate results from the official TensorFlow implementation (TTUR), use the mode="legacy_tensorflow" flag in the fid.compare_folders function. This mode is designed to be equivalent to the official implementation found in the TTUR repository.

Note that the official TTUR implementation is highly sensitive to the TensorFlow version. For compatibility with the original implementation, it is recommended to use tensorflow-gpu==1.14 and python 3.6.

from cleanfid import fid

score = fid.compare_folders("test_fake/", "test_real/", mode="legacy_tensorflow")

To publish the built wheel to PyPI, use twine to upload the file from the dist/ directory.

python -m twine upload dist/clean_fid-0.1.35-py3-none-any.whl