SemHash

SemHash allows you to use any modality (like images) by providing a custom encoder that implements the Encoder protocol. An encoder must have an encode(inputs, **kwargs) method that accepts a batch of inputs and returns a numpy array of embeddings.

Example implementation for images using timm:

class VisionEncoder:
    """Custom encoder using timm models. Implements the Encoder protocol."""

    def __init__(self, model_name: str = "mobilenetv3_small_100.lamb_in1k"):
        self.model = timm.create_model(model_name, pretrained=True, num_classes=0).eval()
        data_config = timm.data.resolve_model_data_config(self.model)
        self.transform = timm.data.create_transform(**data_config, is_training=False)

    def encode(self, inputs, batch_size: int = 128):
        """Encode a batch of PIL images into embeddings."""
        import numpy as np

        # Convert grayscale to RGB if needed
        rgb_inputs = [img.convert("RGB") if img.mode != "RGB" else img for img in inputs]

        # Process in batches to avoid memory issues
        all_embeddings = []
        with torch.no_grad():
            for i in range(0, len(rgb_inputs), batch_size):
                batch_inputs = rgb_inputs[i : i + batch_size]
                batch = torch.stack([self.transform(img) for img in batch_inputs])
                embeddings = self.model(batch).numpy()
                all_embeddings.append(embeddings)

        return np.vstack(all_embeddings)

Use SemHash to deduplicate a second dataset (e.g., test_texts) against an existing reference dataset (e.g., train_texts). This is useful for eliminating train/test leakage.

Unlike the self_ methods, these require passing the target records as an argument:

• deduplicate(records=..., threshold=...): Removes records from the target dataset that are too similar to the reference dataset.
• filter_outliers(records=..., outlier_percentage=...): Identifies outliers in the target dataset relative to the reference.
• find_representative(records=..., selection_size=...): Finds representative samples in the target dataset relative to the reference.

from datasets import load_dataset
from semhash import SemHash

# Load two datasets to deduplicate
train_texts = load_dataset("ag_news", split="train")["text"]
test_texts = load_dataset("ag_news", split="test")["text"]

# Initialize a SemHash instance with the training data
semhash = SemHash.from_records(records=train_texts)

# Deduplicate the test data against the training data, optionally with a specific threshold
deduplicated_test_texts = semhash.deduplicate(records=test_texts, threshold=0.9).selected

# Filter outliers from the test data against the training data, optionally with a specific percentage
filtered_test_texts = semhash.filter_outliers(records=test_texts, outlier_percentage=0.1).selected

# Find representative texts in the test data against the training data, optionally with a specific selection size
representative_test_texts = semhash.find_representative(records=test_texts, selection_size=10).selected

When working with datasets containing multiple fields (e.g., a dictionary or a row from a HuggingFace dataset), you can specify which columns should be used for the semantic hashing by passing a list of strings to the columns parameter in SemHash.from_records.

from datasets import load_dataset
from semhash import SemHash

# Load the dataset
dataset = load_dataset("squad_v2", split="train")

# Convert the dataset to a list of dictionaries
records = [dict(row) for row in dataset]

# Initialize SemHash with the columns to deduplicate
semhash = SemHash.from_records(records=records, columns=["question", "context"])

# Deduplicate the records
deduplicated_records = semhash.self_deduplicate().selected

# Filter outliers from the records
filtered_records = semhash.self_filter_outliers().selected

# Find representative samples in the records
representative_records = semhash.self_find_representative().selected

To compare a new set of records against an existing SemHash instance (e.g., deduplicating a test set against a training set), use the non-self_ prefixed methods. These methods take the new records as an argument and return the results for those specific records.

Common operations include:

• deduplicate(records=...): Deduplicates the provided records against the records used to initialize the SemHash instance.
• filter_outliers(records=...): Filters outliers from the provided records based on the existing instance.
• find_representative(records=...): Finds representative samples within the provided records.

from datasets import load_dataset
from semhash import SemHash

# Load two datasets to deduplicate
train_texts = load_dataset("ag_news", split="train")["text"]
test_texts = load_dataset("ag_news", split="test")["text"]

# Initialize a SemHash instance with the training data
semhash = SemHash.from_records(records=train_texts)

# Deduplicate the test data against the training data
deduplicated_test_texts = semhash.deduplicate(records=test_texts).selected

# Filter outliers from the test data
filtered_test_texts = semhash.filter_outliers(records=test_texts).selected

# Find representative texts in the test data
representative_test_texts = semhash.find_representative(records=test_texts).selected

To process images, you must provide a vision model (e.g., from sentence-transformers).

1. Load a vision model (e.g., clip-ViT-B-32).
2. Initialize SemHash using from_records, specifying the column containing the images and passing the model.
3. Use self_deduplicate(), self_filter_outliers(), or self_find_representative() to process the images.

Note: This requires pip install sentence-transformers.

from datasets import load_dataset
from sentence_transformers import SentenceTransformer
from semhash import SemHash

# Load an image dataset and vision model
model = SentenceTransformer('clip-ViT-B-32')
dataset = load_dataset("uoft-cs/cifar10", split="test")

# Initialize a SemHash instance with the 'img' column
semhash = SemHash.from_records(list(dataset), columns=["img"], model=model)

# Deduplicate the images
deduplicated_images = semhash.self_deduplicate().selected

# Filter outliers
filtered_images = semhash.self_filter_outliers().selected

# Find representative images
representative_images = semhash.self_find_representative().selected

SemHash provides high-level methods for single-dataset operations on text. Text processing works out of the box using fast Model2Vec embeddings.

To perform these operations on a single dataset, use the self_ prefixed methods:

• self_deduplicate(): Removes semantic duplicates from the dataset.
• self_filter_outliers(): Identifies and removes outliers.
• self_find_representative(): Selects representative samples from the dataset.

Note: These examples assume you have the datasets library installed (pip install datasets).

from datasets import load_dataset
from semhash import SemHash

# Load a dataset to deduplicate
texts = load_dataset("ag_news", split="train")["text"]

# Initialize a SemHash instance
semhash = SemHash.from_records(records=texts)

# Deduplicate the texts
deduplicated_texts = semhash.self_deduplicate().selected

# Filter outliers
filtered_texts = semhash.self_filter_outliers().selected

# Find representative texts
representative_texts = semhash.self_find_representative().selected

To perform operations on a single dataset, use the self_* methods on a SemHash instance initialized via SemHash.from_records. These methods return an object where you can access the processed results using the .selected attribute.

Common operations include:

• self_deduplicate(): Removes duplicate entries within the dataset.
• self_filter_outliers(): Removes outlier entries from the dataset.
• self_find_representative(): Identifies and selects representative samples from the dataset.

from datasets import load_dataset
from semhash import SemHash

# Load a dataset to deduplicate
texts = load_dataset("ag_news", split="train")["text"]

# Initialize a SemHash instance
semhash = SemHash.from_records(records=texts)

# Deduplicate the texts
duplicated_texts = semhash.self_deduplicate().selected

# Filter outliers
filtered_texts = semhash.self_filter_outliers().selected

# Find representative texts
representative_texts = semhash.self_find_representative().selected