repository·master·Indexed 26 days ago
https://github.com/idealo/imagededupA Python package for finding exact and near duplicates in image collections. It supports multiple deduplication algorithms, including hashing methods (PHash, DHash, AHash, WHash) and Convolutional Neural Networks (CNN). The library allows for custom PyTorch model integration via CustomModel and provides tools for encoding generation, duplicate visualization, and quality evaluation using metrics like MAP, NDCG, and F1-score.
The package supports several algorithms for finding exact and near duplicates:
When choosing between hashing and CNN methods, consider the following trade-offs:
dhash, phash, etc.) are significantly faster than the cnn method. dhash is typically the fastest.cnn method is much more robust for near-duplicates and transformed images, whereas hashing methods are primarily effective for exact or near-exact matches.cnn method is computationally intensive and will run much faster if a GPU is available. Benchmarks provided are based on CPU-only execution.To use your own neural network architecture, wrap your PyTorch model in a CustomModel object. Your model must be a subclass of torch.nn.Module and implement a forward method that returns a tensor of shape (batch_size, features).
You must provide:
name: A string identifier for the model.model: The PyTorch model instance.transform: A function (e.g., torchvision.transforms.Compose) that converts a PIL.Image into a PyTorch tensor compatible with your model's preprocessing requirements.Note: name and transform do not have to be attributes of the model class itself; they can be passed separately to the CustomModel constructor.
from imagededup.methods import CNN
from imagededup.utils import CustomModel
import torch
from torchvision.transforms import transforms
# Define your custom model
class MyModel(torch.nn.Module):
transform = transforms.Compose(
[
transforms.ToTensor()
]
)
name = 'my_custom_model'
def __init__(self):
super().__init__()
# Define the layers of the model here
def forward(self, x):
# Do something with x
return x
# Wrap in CustomModel
custom_config = CustomModel(name=MyModel.name,
model=MyModel(),
transform=MyModel.transform)
# Initialize CNN with your custom model
cnn = CNN(model_config=custom_config)
# Use the model as usual
# ...You can use high-performance prepackaged models (MobileNetV3, ViT, or EfficientNet) by wrapping them in a CustomModel construct and passing them to the CNN class via the model_config argument.
Available prepackaged models:
MobilenetV3 (Default)ViT (Vision Transformer)EfficientNet (EfficientNet B4)from imagededup.methods import CNN
from imagededup.utils import CustomModel
from imagededup.utils.models import ViT, MobilenetV3, EfficientNet
# Declare a custom config with a prepackaged model
custom_config = CustomModel(name=EfficientNet.name,
model=EfficientNet(),
transform=EfficientNet.transform)
# Pass the config to the CNN object
cnn = CNN(model_config=custom_config)
# Use the model as usual
# ...You can install imagededup via PyPI (recommended) or directly from the GitHub source.
From PyPI:
pip install imagededupFrom GitHub source:
git clone https://github.com/idealo/imagededup.git
cd imagededup
pip install .pip install imagededupAfter generating a duplicate dictionary for an image directory (e.g., using find_duplicates), you can visualize the duplicates associated with a specific file using the plot_duplicates method from imagededup.utils.
This function generates a plot showing the target image alongside its identified duplicates. You can pass either a simple duplicate map (filenames as keys, list of duplicates as values) or a map containing similarity scores.
When evaluating duplicates, imagededup treats symmetric relationships differently depending on the metric type:
Information Retrieval Metrics (map, ndcg, jaccard): These treat each key in the map as an independent 'query'. If a symmetric relationship is missing (e.g., A lists B as a duplicate, but B does not list A), the error is counted twice (once for query A and once for query B).
Classification Metrics (precision, recall, f1-score): These form unique pairs of images and label them as 0 (non-duplicate) or 1 (duplicate). In this mode, a missing symmetric relationship is only accounted for once as a single failed pair.
You can use native-style built-in names (like __builtin_ffs) regardless of which compiler is in use (e.g., in MSVC or older GCC versions) by defining the PSNIP_BUILTIN_EMULATE_NATIVE macro before including builtin.h.
When this macro is defined, the header provides definitions for missing native built-ins, allowing you to write code that is portable across different compilers without changing the function names.
Based on benchmark results, use the following strategies depending on your goal:
Use the cnn method with a min_similarity_threshold between 0.5 and 0.9. Hashing methods generally perform poorly for near-duplicate detection.
Use the cnn method with the default min_similarity_threshold of 0.9. Hashing methods are not effective for most transformations (except resizing).
dhash with a max_distance_threshold of 0 for the fastest performance.cnn method with a high min_similarity_threshold (e.g., 0.9).To build and view the project documentation locally, install mkdocs and mkdocs-material via pip, then use the mkdocs serve command. The documentation will be available at http://127.0.0.1:8000/.
pip install mkdocs mkdocs-material
mkdocs serveTo ensure maximum portability when using builtin.h, you should #include the exact-int module before including the header.
If you prefer not to add the exact-int module to your project, you can omit it; the module will then rely on <stdint.h>. Alternatively, you can manually define the following macros to appropriate types:
psnip_int8_tpsnip_uint8_tpsnip_int16_tpsnip_uint16_tpsnip_int32_tpsnip_uint32_tpsnip_int64_tpsnip_uint64_tTo automatically generate the code documentation for the project, run the autogen.py script using Python.
python autogen.py