DOCSEARCHbyAlgolia
Submit Docs

Byaldi Documentation

repository·main·Indexed 21 days ago

https://github.com/answerdotai/byaldi

A lightweight Python wrapper around the ColPali engine for using late-interaction multi-modal models like ColQwen2. Byaldi provides a RAGatouille-inspired API to index PDFs and images, perform semantic searches, and integrate with Vision Language Models (VLMs) such as Claude for document retrieval and RAG pipelines.

Tokens
2.8K
Snippets
14
Records
14
Agent score
73%

What's inside Byaldi

  1. Install Byaldi and its dependencies

    main

    Byaldi is a wrapper around ColPali for multi-modal retrieval. To use it, you must install system-level dependencies and recommended Python packages.

    1. Install Poppler

    Poppler is required for pdf2image to convert PDFs to images.

    • MacOS (Homebrew): brew install poppler
    • Debian/Ubuntu: sudo apt-get install -y poppler-utils

    2. Install Byaldi and Flash-Attention

    It is recommended to install flash-attn for optimal performance with models like Gemma.

    pip install --upgrade byaldi
pip install flash-attn

    Hardware Note

    Encoding documents is computationally intensive. While older GPUs work, using a CPU or MPS will result in poor performance during the encoding phase.

    pip install --upgrade byaldi
pip install flash-attn

  2. Install Byaldi and dependencies

    main

    To use Byaldi for PDF processing, you need to install byaldi, claudette (for Claude integration), and pdf2images. pdf2images requires a system-level installation of poppler.

    Python packages:

    !pip install byaldi claudette

    System dependencies (Poppler):

    • MacOS: brew install poppler
    • Linux: sudo apt-get install poppler-utils
    • Windows: Follow instructions from Poppler-Windows.

  3. Chat with your PDF using Byaldi and Claude

    main

    To build a RAG pipeline that answers questions using visual context, follow these steps:

    1. Index the PDF using RAGMultiModalModel.index with store_collection_with_index=True.
    2. Search for the relevant page using RAG.search.
    3. Decode the base64 image from the search result into bytes.
    4. Query Claude using the claudette library, passing both the image bytes and the text query.

    Full Workflow Example:

    import base64
import os
from byaldi import RAGMultiModalModel
from claudette import *

# 1. Setup
os.environ["HF_TOKEN"] = "YOUR_HF_TOKEN"
os.environ["ANTHROPIC_API_KEY"] = "YOUR_ANTHROPIC_API_KEY"

# 2. Load and Index
RAG = RAGMultiModalModel.from_pretrained("vidore/colpali-v1.2", verbose=1)
RAG.index(
    input_path="./docs/attention.pdf",
    index_name="attention",
    store_collection_with_index=True,
    overwrite=True
)

# 3. Search
query = "What's the BLEU score for the transformer base model?"
results = RAG.search(query, k=1)

# 4. Pass to Claude
image_bytes = base64.b64decode(results[0].base64)
chat = Chat(models[1]) # models[1] is Claude Sonnet 3.5
print(chat([image_bytes, query]))
    import base64
import os
from byaldi import RAGMultiModalModel
from claudette import *

os.environ["HF_TOKEN"] = "YOUR_HF_TOKEN"
os.environ["ANTHROPIC_API_KEY"] = "YOUR_ANTHROPIC_API_KEY"

RAG = RAGMultiModalModel.from_pretrained("vidore/colpali-v1.2", verbose=1)
RAG.index(
    input_path="./docs/attention.pdf",
    index_name="attention",
    store_collection_with_index=True,
    overwrite=True
)

query = "What's the BLEU score for the transformer base model?"
results = RAG.search(query, k=1)

image_bytes = base64.b64decode(results[0].base64)
chat = Chat(models[1])
print(chat([image_bytes, query]))

  4. Create a new index with index()

    main

    The .index() method allows you to create a searchable index from a single PDF, a single image, or a directory of documents.

    Key Parameters:

    • input_path: Path to your documents (file or directory).
    • index_name: The name of the index. It will be saved at {index_root}/{index_name}/.
    • store_collection_with_index (bool): If True, the base64-encoded versions of documents are stored in the index. This allows immediate use with LLMs but increases storage and memory usage. Defaults to False.
    • doc_ids (list[int]): Optional list of integer IDs for documents. Must match the number of documents provided.
    • metadata (list[dict]): Optional list of dictionaries containing metadata for each document. Must match the number of documents.
    • overwrite (bool): If True, overwrites an existing index of the same name.
    from byaldi import RAGMultiModalModel

RAG = RAGMultiModalModel.from_pretrained("vidore/colqwen2-v1.0")
RAG.index(
    input_path="docs/",
    index_name="my_index",
    store_collection_with_index=False,
    doc_ids=[0, 1, 2],
    metadata=[{"author": "John Doe", "date": "2021-01-01"}],
    overwrite=True
)

  5. Add documents to an existing index

    main

    Because Byaldi indexes are in-memory, you can easily ingest new documents into an existing index using the .add_to_index() method. This method accepts parameters similar to the original .index() method.

    RAG.add_to_index(
    "path_to_new_docs",
    store_collection_with_index=False
)

  6. Search an index with search()

    main

    Once an index is loaded, use .search(query, k=N) to retrieve the most relevant document segments.

    Return Value: A list of Result objects (which can also be treated as dictionaries) sorted by relevance score.

    Result Schema:

    • doc_id: 0-indexed integer.
    • page_num: 1-indexed integer (useful for PDF manipulation).
    • score: The relevance score.
    • metadata: Dictionary of metadata (if provided during indexing).
    • base64: The base64 encoded document (only if store_collection_with_index=True was used during indexing).
    results = RAG.search("your query text", k=3)

# Example result structure:
# [
#     {
#         "doc_id": 0,
#         "page_num": 10,
#         "score": 12.875,
#         "metadata": {},
#         "base64": None
#     }
# ]

  7. Load a model or an existing index with RAGMultiModalModel

    main

    Use RAGMultiModalModel to either initialize a new model from a checkpoint or load a previously created index.

    Load from checkpoint

    Use .from_pretrained(model_name) to download and load a specific model (e.g., vidore/colqwen2-v1.0).

    Load from existing index

    Use .from_index(index_name) to load a model and its associated index from disk. By default, Byaldi looks in the .byaldi/ directory.

    from byaldi import RAGMultiModalModel

# Load a new model
RAG = RAGMultiModalModel.from_pretrained("vidore/colqwen2-v1.0")

# Load an existing index
RAG = RAGMultiModalModel.from_index("your_index_name")

  8. Index a PDF document with Byaldi

    main

    The index method creates representations of a document (like a PDF) and stores them.

    Arguments:

    • input_path: Path to the document.
    • index_name: A unique name for the index.
    • store_collection_with_index: If True, stores the image representation of pages as base64 strings in the results, making it easy to pass to Vision Language Models (VLMs).
    • overwrite: If True, replaces any existing index with the same name.
    RAG.index(
    input_path="./docs/attention.pdf",
    index_name="attention",
    store_collection_with_index=True,
    overwrite=True
)

  9. Initialize RAGMultiModalModel from a pretrained model

    main

    To start using Byaldi, initialize the RAGMultiModalModel using the from_pretrained method. You can specify a model checkpoint (e.g., vidore/colqwen2-v1.0). If the model requires authentication, ensure your Hugging Face token is set in the HF_TOKEN environment variable.

    import os
from byaldi import RAGMultiModalModel

# os.environ["HF_TOKEN"] = "YOUR_HF_TOKEN"

model = RAGMultiModalModel.from_pretrained("vidore/colqwen2-v1.0")

  11. Search an existing index with `search()`

    main

    Perform semantic searches against an index using the search() method.

    Parameters:

    • query: The text string to search for.
    • k: The number of top results to return.
    • filter_metadata: (Optional) A dictionary used to filter results based on metadata keys and values.

    Returns: A list of result objects. Each result contains:

    • doc_id: Unique identifier for the document.
    • page_num: The page number where the match was found.
    • score: The similarity score.
    • base64: (Optional) The base64 encoded image of the page, available if store_collection_with_index=True was used during indexing.
    # Basic search
results = model.search("what's the BLEU score of this new strange method?", k=5)

# Search with metadata filtering
results = model.search("query", k=5, filter_metadata={"filename": "attention.pdf"})

for result in results:
    print(f"Doc ID: {result.doc_id}, Page: {result.page_num}, Score: {result.score}")

  12. Search an index for relevant pages

    main

    Use the search method to perform a textual query against an existing index. It returns the top k pages that match the query.

    query = "What's the BLEU score for the transformer base model?"
results = RAG.search(query, k=1)

# Accessing result attributes:
# results[0].page_num -> The page number
# results[0].base64 -> The base64 encoded image of the page (if store_collection_with_index=True)
    results = RAG.search(query, k=1)