RAGatouille Documentation

RAGatouille focuses on Late-Interaction retrieval (e.g., ColBERT), which uses a bag-of-embeddings approach. This differs from traditional methods in the following ways:

Why Late-Interaction works: Instead of compressing a whole document into a single vector (which limits information capacity), ColBERT breaks documents into a 'bag' of contextualized units of information. This allows the model to capture meaning at the token level, making it more robust to different phrasing and improving generalization to unseen data.

To fine-tune an existing ColBERT model, instantiate RAGTrainer with a pretrained_model_name that points to a ColBERT instance. If pretrained_model_name points to a different transformer, it will initialize a new ColBERT model from those weights.

Use prepare_training_data() to process your pairs and corpus, then call train() to start the process.

from ragatouille import RAGTrainer
from ragatouille.utils import get_wikipedia_page

pairs = [
    ("What is the meaning of life ?", "The meaning of life is 42"),
    ("What is Neural Search?", "Neural Search is a terms referring to a family of ..."),
    # You need many more pairs to train!
    ...
]

my_full_corpus = [get_wikipedia_page("Hayao_Miyazaki"), get_wikipedia_page("Studio_Ghibli")]

trainer = RAGTrainer(model_name = "MyFineTunedColBERT",
        pretrained_model_name = "colbert-ir/colbertv2.0") # In this example, we run fine-tuning

# This step handles all the data processing
trainer.prepare_training_data(raw_data=pairs,
                                data_out_path="./data/",
                                all_documents=my_full_corpus)

trainer.train(batch_size=32) # Train with the default hyperparams

You can fine-tune ColBERT models without manual annotations by using the instructor library and an LLM (like OpenAI) to generate synthetic [query, relevant_passage] pairs. This process involves:

1. Extracting structured queries: Use instructor.patch(OpenAI(...)) and a Pydantic model to force the LLM to return specific query types (e.g., hypothetical questions or search queries) based on document chunks.
2. Creating training pairs: Map the generated queries to their corresponding document chunks.
3. Preparing training data: Use RAGTrainer.prepare_training_data to combine these pairs with your full corpus to automatically mine hard negatives.

This approach allows for domain adaptation without the cost of human labeling.

import instructor
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import List
from ragatouille import RAGTrainer

# 1. Setup Instructor with OpenAI
client = instructor.patch(OpenAI(api_key=os.environ["OPENAI_API_KEY"]))

# 2. Define schema for synthetic query generation
class QueryForPassage(BaseModel):
    hypothetical_questions: List[str] = Field(
        default_factory=list,
        description="A wide variety of hypothetical questions that this document could answer.",
    )
    hypothetical_queries: List[str] = Field(
        default_factory=list,
        description="A wide variety of hypothetical queries that this document would be relevant to.",
    )

# 3. Generate queries using the LLM
# (Assuming 'relevant_documents' is a list of text chunks)
candidate_queries = []
for doc in relevant_documents:
    candidate = client.chat.completions.create(
        model="gpt-4-1106-preview",
        response_model=QueryForPassage,
        messages=[
            {"role": "system", "content": "You are an expert AI..."},
            {"role": "user", "content": doc},
        ],
    )
    candidate_queries.append(candidate)

# 4. Format pairs for RAGatouille
pairs = []
for candidates, doc in zip(candidate_queries, relevant_documents):
    candidates_dict = candidates.model_dump()
    # Combine different query types into the training pairs
    queries = candidates_dict['hypothetical_questions'] + candidates_dict['hypothetical_queries']
    for q in queries:
        pairs.append([q, doc])

# 5. Prepare training data with hard negative mining
trainer = RAGTrainer(model_name="MyModel", pretrained_model_name="colbert-ir/colbertv2.0")
trainer.prepare_training_data(
    raw_data=pairs,
    all_documents=documents,
    num_new_negatives=10,
    mine_hard_negatives=True,
)

You can create an index by loading a pretrained model or your own trained model using RAGPretrainedModel.from_pretrained().

When calling .index(), you can provide:

• index_name: The name of the index.
• collection: A list of documents to index.
• document_ids (optional): A list of IDs corresponding to the documents.
• document_metadatas (optional): A list of dictionaries containing metadata for each document.

from ragatouille import RAGPretrainedModel
from ragatouille.utils import get_wikipedia_page

RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
my_documents = [get_wikipedia_page("Hayao_Miyazaki"), get_wikipedia_page("Studio_Ghibli")]
index_path = RAG.index(index_name="my_index", collection=my_documents)

# Indexing with IDs and Metadata
document_ids = ["miyazaki", "ghibli"]
document_metadatas = [
    {"entity": "person", "source": "wikipedia"},
    {"entity": "organisation", "source": "wikipedia"},
]
index_path = RAG.index(
    index_name="my_index_with_ids_and_metadata",
    collection=my_documents,
    document_ids=document_ids,
    document_metadatas=document_metadatas,
)