FashionCLIP Documentation

To use the internal FCLIPDataset and FashionCLIP abstractions, install the package in editable mode from the project root.

$ pip install -e .

You can quickly install the fashion-clip package using pip to access the high-level API for generating embeddings.

$ pip install fashion-clip

To use FashionCLIP, you must first wrap your product data in an FCLIPDataset. The dataset expects a catalog which is a list of dictionaries. Each dictionary representing a product must contain the following keys:

• id: A unique identifier for the product.
• image: The filename of the image.
• caption: A text description of the product.

Currently, images are assumed to be stored in a local folder, though the API supports s3 via image_source_type.

# Example: Loading a local dataset
catalog = [
    {'id': 1, 'image': '16867424.jpg', 'caption': 'light red polo shirt'},
    {'id': 2, 'image': '16790484.jpg', 'caption': 'an adidas sneaker'},
    {'id': 3, 'image': '16198646.jpg', 'caption': 'dark red polo shirt'},
]

dataset = FCLIPDataset(
    'farfetch_local',
    image_source_path='./images',
    image_source_type='local',
    catalog=catalog
)

Since FashionCLIP is hosted on Hugging Face, you can use the standard transformers library to load the model and processor for zero-shot tasks like image-text similarity scoring.

from PIL import Image
import requests
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("patrickjohncyh/fashion-clip")
processor = CLIPProcessor.from_pretrained("patrickjohncyh/fashion-clip")

image = Image.open("images/image1.jpg")

inputs = processor(text=["a photo of a red shoe", "a photo of a black shoe"],
                   images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)  
print(probs)

Use the FashionCLIP class to encode lists of images and texts into embeddings. To perform similarity comparisons (like dot product), you should manually normalize the resulting embeddings to unit norm using np.linalg.norm.

from fashion_clip.fashion_clip import FashionCLIP
import numpy as np

fclip = FashionCLIP('fashion-clip')

# we create image embeddings and text embeddings
image_embeddings = fclip.encode_images(images, batch_size=32)
text_embeddings = fclip.encode_text(texts, batch_size=32)

# we normalize the embeddings to unit norm (so that we can use dot product instead of cosine similarity to do comparisons)
image_embeddings = image_embeddings/np.linalg.norm(image_embeddings, ord=2, axis=-1, keepdims=True)
text_embeddings = text_embeddings/np.linalg.norm(text_embeddings, ord=2, axis=-1, keepdims=True)

Use fclip.display_attention to visualize which parts of an image the model is focusing on for a specific text query. This is useful for debugging or understanding model behavior.

fclip.display_attention(
    image_path='images/nike_dress.jpg',
    query_text='nike',
    pixel_size=15,
    iterations=15
)

Use fclip.retrieval to find products in your initialized FCLIPDataset that match a text query. The method returns a list of candidate indices from the dataset.

# Retrieve products matching the query 'shoes'
candidates = fclip.retrieval(['shoes'])
print(candidates)

# Display the results using the dataset object
# dataset.ids[candidates[0]] retrieves the actual IDs for the top match
_ = dataset.display_products(dataset.ids[candidates[0]], fields=tuple(['id']))

Create a FashionCLIP instance by providing a model identifier and an FCLIPDataset instance.

Model options:

• A pre-trained model name (e.g., 'fashion-clip' or 'openai/clip-vit-base-patch32').
• A path to a local model file.

Note on performance: If the specific combination of model and dataset has been processed before, FashionCLIP will download pre-processed vectors via hashing. Otherwise, it will generate new vectors for the dataset upon instantiation.

from fashion_clip.fashion_clip import FashionCLIP

# Instantiate with a pre-trained model and your dataset
fclip = FashionCLIP('fashion-clip', dataset)

Use fclip.zero_shot_classification to classify one or more images against a list of text labels. This method does not rely on pre-processed vectors from the dataset but performs inference on the provided image paths.

test_captions = ["nike sneakers", "adidas sneakers", "converse", "a gucci dress"]
test_img_path = 'images/16790484.jpg'

# Returns classification results for the provided image paths
fclip.zero_shot_classification([test_img_path], test_captions)