ColPali

ColPali is supported by various community-developed libraries and integrations for vector databases and deployment frameworks.

Key Libraries

• Byaldi: A high-level library (equivalent to RAGatouille for ColPali) that leverages colpali-engine to facilitate indexing and storing embeddings.
• PyVespa: Enables interaction with the Vespa vector database with detailed ColPali support.
• EmbedAnything: Provides end-to-end ColPali inference using Candle and ONNX backends.
• BentoML: Facilitates easy deployment of ColPali with features like adaptive batching and zero-copy I/O.
• ColiVara: A web-first retrieval API for storing and searching documents based on visual embeddings.
• Astra-multivector: Provides enterprise-grade integration with AstraDB, implementing token pooling and embedding caching.
• Mixpeek: A production platform for multimodal late-interaction retrieval supporting ColBERT, ColPaLI, and ColQwen2.
• NoOCR: An open-source end-to-end solution for complex PDFs powered by ColPali.

Supported Vector Databases

• Vespa (via PyVespa)
• Qdrant
• Elastic Search
• Weaviate
• Milvus
• AstraDB (via Astra-multivector)
• LanceDB

To visualize which image patches are most salient to specific query terms, install the interpretability extension and use the interpretability module. You will need to generate image and query embeddings, calculate the number of patches with processor.get_n_patches, and create an image_mask with processor.get_image_mask before calling get_similarity_maps_from_embeddings and plot_all_similarity_maps.

pip install colpali-engine[interpretability]

import torch
from PIL import Image

from colpali_engine.interpretability import (
    get_similarity_maps_from_embeddings,
    plot_all_similarity_maps,
)
from colpali_engine.models import ColPali, ColPaliProcessor
from colpali_engine.utils.torch_utils import get_torch_device

model_name = "vidore/colpali-v1.3"
device = get_torch_device("auto")

# Load the model
model = ColPali.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map=device,
).eval()

# Load the processor
processor = ColPaliProcessor.from_pretrained(model_name)

# Load the image and query
image = Image.open("shift_kazakhstan.jpg")
query = "Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?"

# Preprocess inputs
batch_images = processor.process_images([image]).to(device)
batch_queries = processor.process_queries([query]).to(device)

# Forward passes
with torch.no_grad():
    image_embeddings = model.forward(**batch_images)
    query_embeddings = model.forward(**batch_queries)

# Get the number of image patches
n_patches = processor.get_n_patches(image_size=image.size, patch_size=model.patch_size)

# Get the tensor mask to filter out the embeddings that are not related to the image
image_mask = processor.get_image_mask(batch_images)

# Generate the similarity maps
batched_similarity_maps = get_similarity_maps_from_embeddings(
    image_embeddings=image_embeddings,
    query_embeddings=query_embeddings,
    n_patches=n_patches,
    image_mask=image_mask,
)

# Get the similarity map for our (only) input image
similarity_maps = batched_similarity_maps[0]  # (query_length, n_patches_x, n_patches_y)

# Tokenize the query
query_tokens = processor.tokenizer.tokenize(query)

# Plot and save the similarity maps for each query token
plots = plot_all_similarity_maps(
    image=image,
    query_tokens=query_tokens,
    similarity_maps=similarity_maps,
)
for idx, (fig, ax) in enumerate(plots):
    fig.savefig(f"similarity_map_{idx}.png")

A variety of notebooks and cookbooks are available for learning how to use ColPali in different RAG (Retrieval-Augmented Generation) scenarios:

• General Cookbooks: Tony's Cookbooks (ILLUIN)
• Vision RAG: Manu's Vision Rag Tutorial
• Database Specific:
  • Indexing with Qdrant
  • Weaviate Tutorial
  • Multi-Modal Retrieval with Milvus
• Advanced RAG:
  • Multimodal RAG with ColQwen2 and Amazon Nova
  • Finance Report Analysis with Gemini

To perform document retrieval using ColQwen2, load the model and processor, process your images and queries, and perform a forward pass to obtain embeddings. Use processor.score_multi_vector to calculate similarity scores between query and image embeddings.

import torch
from PIL import Image
from transformers.utils.import_utils import is_flash_attn_2_available

from colpali_engine.models import ColQwen2, ColQwen2Processor

model_name = "vidore/colqwen2-v1.0"

model = ColQwen2.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",  # or "mps" if on Apple Silicon
    attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None,
).eval()

processor = ColQwen2Processor.from_pretrained(model_name)

# Your inputs
images = [
    Image.new("RGB", (128, 128), color="white"),
    Image.new("RGB", (64, 32), color="black"),
]
queries = [
    "What is the organizational structure for our R&D department?",
    "Can you provide a breakdown of last year’s financial performance?",
]

# Process the inputs
batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_queries(queries).to(model.device)

# Forward pass
with torch.no_grad():
    image_embeddings = model(**batch_images)
    query_embeddings = model(**batch_queries)

scores = processor.score_multi_vector(query_embeddings, image_embeddings)