LocalGPT

The IndexingPipeline follows a sequential processing pattern designed for efficient memory usage and progress tracking. It processes documents through several distinct stages in order. If a stage is optional (like contextual enrichment or graph extraction), it is skipped if the corresponding component is not provided.

The pipeline stages are:

1. Document Processing & Chunking: Converting files to text and splitting them into manageable pieces.
2. Contextual Enrichment (optional): Using an LLM to add context to chunks.
3. Dense Indexing: Generating embeddings and storing them in a vector database.
4. Graph Extraction (optional): Extracting relationships for graph-based retrieval.

def run(self, file_paths: List[str]):
    with timer("Complete Indexing Pipeline"):
        # Stage 1: Document Processing & Chunking
        all_chunks = []
        doc_chunks_map = {}
        
        # Stage 2: Contextual Enrichment (optional)
        if self.contextual_enricher:
            all_chunks = self.contextual_enricher.enrich_batch(all_chunks)
        
        # Stage 3: Dense Indexing (embedding + storage)
        if self.vector_indexer:
            self.vector_indexer.index_chunks(all_chunks, table_name)
        
        # Stage 4: Graph Extraction (optional)
        if self.graph_extractor:
            self.graph_extractor.extract_and_store(all_chunks)

The system is organized into several functional modules that orchestrate the flow from raw PDF to a human-readable answer.

Key Modules

• main.py: The entry point that configures models and orchestrates pipelines.
• rag_system/pipelines/: High-level orchestration.
  • indexing_pipeline.py: Converts raw PDFs into searchable data.
  • retrieval_pipeline.py: Handles query processing, retrieval, and answer generation.
• rag_system/indexing/: Data processing and storage.
  • multimodal.py: Extracts text/images and generates embeddings (using colqwen2-v1.0).
  • representations.py: Defines text embedding models (e.g., Qwen2-7B-instruct).
  • embedders.py: Manages LanceDB vector database connections.
• rag_system/retrieval/: Search and ranking.
  • retrievers.py: Searches LanceDB for text and image chunks.
  • reranker.py: Contains QwenReranker for relevance ordering.
• rag_system/agent/: The user interaction loop.
• rag_system/utils/: Utility clients like OllamaClient.

Data Flow

1. Indexing: MultimodalProcessor splits PDFs $\rightarrow$ extracts text and page images $\rightarrow$ QwenEmbedder (text) and LocalVisionModel (image) generate embeddings $\rightarrow$ VectorIndexer stores them in LanceDB tables.
2. Retrieval: User query $\rightarrow$ MultiVectorRetriever searches LanceDB $\rightarrow$ QwenReranker re-orders results $\rightarrow$ Top results (text + images) passed to VLM (qwen-vl) $\rightarrow$ VLM extracts facts $\rightarrow$ llama3 synthesizes the final answer.

The Retrieval Pipeline is responsible for taking a user query and indexed tables to retrieve relevant text chunks and synthesize an answer. It follows a multi-stage flow:

1. Query Pre-processing: Uses a QueryTransformer (e.g., HyDEGenerator, GraphQueryTranslator) to expand or rewrite the raw query.
2. Retrieval: Executes search via BM25Retriever, DenseRetriever, or a HybridRetriever against LanceDB vector and Full-Text Search (FTS) indices.
3. Reranking: Optionally uses a reranker (like ColBERTSmall) to re-order results based on relevance.
4. Synthesis: An LLM (via Ollama) uses the top-K chunks to generate and stream a final answer with sources.

The pipeline is exposed via the RetrievalPipeline.answer_stream() iterator, which is designed to be consumed by an SSE (Server-Sent Events) API.

flowchart LR
    Q["User Query"] --> XT["Query Transformer"]
    XT -->|variants| RETRIEVE
    subgraph Retrieval
        RET_BM25[BM25] --> MERGE
        RET_DENSE[Dense Vector] --> MERGE
        style RET_BM25 fill:#444,stroke:#ccc,color:#fff
        style RET_DENSE fill:#444,stroke:#ccc,color:#fff
    end
    MERGE --> RERANK
    RERANK --> K[["Top-K Chunks"]
    K --> SYNTH["Answer Synthesiser\n(LLM)"]
    SYNTH --> A["Answer + Sources"]

LocalGPT follows a client-server architecture designed for private document intelligence. The system is composed of a Next.js frontend, a Python backend, and a core rag_system package that manages the RAG (Retrieval-Augmented Generation) lifecycle.

High-Level Data Flow

1. Frontend Interaction: Users interact with the Next.js UI. API requests are dispatched via src/lib/api.ts using HTTP/JSON.
2. Backend Processing: backend/server.py receives requests, handles CORS, and proxies them to the rag_system core.
3. Agent Decision (Triage): The Agent Loop (rag_system/agent/loop.py) uses a Triage system to decide between performing RAG or providing a direct LLM answer.
4. RAG Execution: If RAG is triggered, the Retrieval Pipeline (pipelines/retrieval_pipeline.py) fetches candidates from LanceDB (using BM25 and dense vectors), an AI Reranker (HuggingFace) sorts them, and the Answer Synthesiser uses Ollama to generate the response.
5. Verification: An optional Verifier performs grounding checks on the generated answers.
6. Indexing: An offline pipeline chunks and embeds uploaded files (e.g., PDFs) into LanceDB.

Storage and Model Providers

• LanceDB: Stores vector tables (chunks and embeddings).
• SQLite: Manages chat history and metadata.
• Ollama: Hosts local LLMs (e.g., qwen3).
• HuggingFace: Provides hosted embedding and reranker models.

The LocalGPT RAG (Retrieval-Augmented Generation) system is a multimodal question-answering system that processes both text and visual layouts. It uses an agentic workflow to decompose complex questions, triage queries, and verify answers against retrieved context.

The system operates through two primary pipelines:

1. Indexing Pipeline: Converts documents into a searchable knowledge base. It extracts text using PyMuPDF, generates embeddings using Qwen/Qwen3-Embedding-0.6B, extracts entities/relationships via a GraphExtractor (using qwen2.5vl:7b) to build a .gml knowledge graph, and stores everything in a LanceDB database.
2. Retrieval Pipeline: Answers queries using an agentic flow. It performs Triage (deciding if retrieval is needed), Query Decomposition (breaking down complex queries via QueryDecomposer), Retrieval (using MultiVectorRetriever and GraphRetriever), Verification (checking context sufficiency via a Verifier), and Synthesis (generating the final answer).