PageIndex

PageIndex is a vectorless, reasoning-based RAG (Retrieval-Augmented Generation) system designed for long, complex professional documents (e.g., legal, financial, technical).

Unlike traditional RAG that relies on vector similarity search and artificial text chunking, PageIndex uses a two-step process that mimics human expert navigation:

1. Tree Index Generation: It transforms a document into a hierarchical tree structure index (similar to a semantic Table of Contents).
2. Reasoning-based Retrieval: It uses LLMs to perform agentic tree search over that index. This allows the model to reason its way to relevant sections based on context, rather than just finding semantically similar text fragments.

Key Advantages:

• No Vector DB: Uses document structure and reasoning instead of vector similarity.
• No Chunking: Organizes documents into natural, semantic sections.
• Traceability: Results are grounded in explicit page and section references.
• Context-Aware: Retrieval can incorporate conversation history and domain knowledge.

A vision-based, vectorless RAG pipeline uses PageIndex to navigate document hierarchies without traditional OCR or vector databases. The workflow follows these steps:

1. Tree Generation: Submit a PDF to PageIndex to build a hierarchical tree of nodes (titles, summaries, and page ranges).
2. Reasoning-based Retrieval: Instead of vector similarity, pass the document tree structure and a user query to a Vision-Language Model (VLM). Ask the VLM to identify relevant node_ids based on the tree's summaries.
3. Visual Context Extraction: Use the retrieved node_ids to find the corresponding page numbers via a node map. Extract these specific pages as images (e.g., using PyMuPDF).
4. Answer Generation: Pass the user query and the extracted page images to a multimodal VLM (like GPT-4o) to generate an answer based purely on visual context.

# Conceptual workflow summary
# 1. Generate Tree
doc_id = pi_client.submit_document(pdf_path)["doc_id"]
tree = pi_client.get_tree(doc_id, node_summary=True)['result']

# 2. Reasoning Retrieval (via VLM)
# Prompt VLM with tree structure to get node_list
# ...

# 3. Visual Context
# Extract images for retrieved nodes
# ...

# 4. Final Answer
# answer = await call_vlm(query, retrieved_page_images)

For documents covering diverse topics, you can combine vector-based semantic search with PageIndex to improve retrieval accuracy. Instead of retrieving raw chunks, use vector search to identify relevant documents first, then use PageIndex to perform structured retrieval within those documents.

The Semantic Search Pipeline

1. Chunking and Embedding: Divide your documents into chunks. Use an embedding model to convert these chunks into vectors and store them in a vector database alongside their corresponding doc_id.
2. Vector Search: For a given query, perform a vector search to retrieve the top-K most relevant chunks and their associated doc_ids.
3. Compute Document Score: Calculate a relevance score for each document to prioritize those with high-quality matches. Use the following formula to aggregate chunk scores while preventing large documents from dominating purely by volume:

$$\text{DocScore}=\frac{1}{\sqrt{N+1}}\sum_{n=1}^N \text{ChunkScore}(n)$$

Where:

• $N$ is the number of content chunks associated with the document.
• $\text{ChunkScore}(n)$ is the relevance score of chunk $n$.
• The $\sqrt{N+1}$ denominator provides diminishing returns, favoring documents with fewer, highly relevant chunks over documents with many weakly relevant ones.

4. Retrieve with PageIndex: Select the documents with the highest DocScore and use their doc_id to perform structured retrieval via the PageIndex retrieval API.

Unlike vector-based RAG, PageIndex allows you to incorporate domain-specific rules or user preferences by injecting them directly into the LLM tree search prompt. This makes node search more targeted without requiring embedding model fine-tuning.

Implementation Pipeline

1. Preference Retrieval: Retrieve relevant user preferences or expert knowledge snippets (e.g., via keyword matching or semantic similarity) based on the incoming query.
2. Enhanced Tree Search: Include the retrieved preference in the prompt alongside the query and the PageIndex_Tree.

Example Prompt Pattern:

prompt = f"""
You are given a question and a tree structure of a document.
You need to find all nodes that are likely to contain the answer.

Query: {query}

Document tree structure:  {PageIndex_Tree}

Expert Knowledge of relevant sections: {Preference}

Reply in the following JSON format:
{{
  "thinking": <reasoning about which nodes are relevant>,
  "node_list": [node_id1, node_id2, ...]
}}
"""

Example Preference Input:

"If the query mentions EBITDA adjustments, prioritize Item 7 (MD&A) and footnotes in Item 8 (Financial Statements) in 10-K reports."