DOCSEARCHbyAlgolia
Submit Docs

RAG Techniques: Advanced Retrieval-Augmented Generation

repository·main·Indexed 12 days ago

https://github.com/nirdiamant/rag_techniques

A collection of 42+ runnable Jupyter notebooks and tutorials for moving RAG prototypes to production. Covers foundational concepts, query enhancement (HyDE, HyPE), context enrichment, advanced retrieval (Graph RAG, RAPTOR, Self-RAG), and evaluation frameworks like Open-RAG-Eval and DeepEval. Includes implementations for LangChain and LlamaIndex.

Tokens
50.4K
Snippets
150
Records
216
Agent score
97%

What's inside RAG Techniques

  1. Overview of Advanced RAG Techniques

    main
    nirdiamant/rag_techniques is a community-driven repository containing over 42 runnable Jupyter notebooks. It covers a wide spectrum of Retrieval-Augmented Generation (RAG) techniques, ranging from foundational concepts to cutting-edge research. The project provides the intuition, implementation code, and references necessary to build accurate, context-rich retrieval systems for production environments.

  2. Key Features of RAG Techniques

    main

    The rag_techniques repository provides resources for elevating Retrieval-Augmented Generation (RAG) systems through several core offerings:

    • State-of-the-art RAG enhancements: Access to advanced techniques to improve retrieval and generation performance.
    • Comprehensive documentation: Detailed explanations for each specific RAG technique implemented.
    • Practical implementation guidelines: Guidance on how to apply these techniques in real-world scenarios.
    • Regular updates: Continuous integration of the latest advancements in the RAG field.

  3. Explore Advanced RAG Techniques via Notebooks and Videos

    main

    The all_rag_techniques directory contains a collection of Jupyter notebooks, each providing a written walkthrough of a specific Retrieval-Augmented Generation (RAG) technique.

    For several key techniques, video explainers are available to provide intuition before diving into the code. These videos cover topics such as:

    • Why AI gets documents wrong: Understanding chunk overlap and 'meaning space'.
    • Searching spreadsheets by meaning: Techniques for turning table rows into searchable semantic lines.
    • Preventing hallucinations: Implementing checkpoints to catch bad chunks or unsupported claims.
    • Embedding granularity: Understanding why paragraph-level embeddings can fail when seeking specific facts.

  4. Explore RAG Techniques by Category

    main

    The rag_techniques repository provides a comprehensive collection of Jupyter notebooks demonstrating various Retrieval-Augmented Generation (RAG) strategies. Techniques are organized into the following functional categories:

    • Foundational 🌱: Basic RAG, RAG with CSV Files, Reliable RAG, Optimizing Chunk Sizes, and Proposition Chunking.
    • Query Enhancement 🔍: Query Transformations, HyDE (Hypothetical Document Embedding), and HyPE (Hypothetical Prompt Embedding).
    • Context Enrichment 📚: Contextual Chunk Headers, Relevant Segment Extraction, Context Window Enhancement, Semantic Chunking, Contextual Compression, and Document Augmentation.
    • Advanced Retrieval 🚀: Fusion Retrieval, Reranking, Multi-faceted Filtering, Hierarchical Indices, Dartboard Retrieval, and Multi-modal RAG with Captioning.
    • Iterative Techniques 🔁: Retrieval with Feedback Loop and Adaptive Retrieval.
    • Evaluation 📊: DeepEval and GroUSE.

    You can access each technique via GitHub notebooks, Google Colab for interactive execution, or video tutorials where available.

  5. Explore Advanced RAG Architectures and Evaluation Techniques

    main

    This repository provides a curated collection of advanced Retrieval-Augmented Generation (RAG) techniques, organized by category. You can access implementation details via Jupyter Notebooks on GitHub or run them directly in Google Colab.

    Advanced Architectures

    • Graph RAG with LangChain: Implementing graph-based retrieval using LangChain.
    • Microsoft GraphRAG: Utilizing Microsoft's GraphRAG approach.
    • RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval.
    • Agentic RAG with Contextual AI: Using agents to drive the RAG process.
    • Self-RAG: A framework for self-improving RAG systems.
    • Corrective RAG (CRAG): Implementing corrective mechanisms to improve retrieval quality.
    • Local Graph RAG with Verifiable Attribution: Graph RAG focused on verifiable source attribution.

    Evaluation Techniques

    • End-to-End RAG Evaluation: Methods for assessing the entire RAG pipeline.
    • Open-RAG-Eval: An evaluation framework for RAG systems.

    Specialized Techniques

    • Explainable Retrieval: Techniques to make the retrieval process transparent.
    • MemoRAG: Advanced memory-based RAG techniques.
    • Sophisticated Controllable Agent: Implementation of highly controllable RAG agents.

  6. Explore Advanced RAG Techniques

    main

    The rag_techniques repository provides a collection of cutting-edge Retrieval-Augmented Generation (RAG) techniques implemented via Jupyter notebooks. The library currently contains over 42 notebooks covering various advanced methodologies to improve RAG system performance.

    Recently added techniques include:

    • MemoRAG: Memory-augmented retrieval.
    • End-to-End RAG Evaluation: Methods for assessing the full RAG pipeline.
    • Open-RAG-Eval: An evaluation framework for RAG.
    • JSON RAG: Techniques for structured data retrieval and generation.

  7. Implement Multi-model RAG with Multimedia Captioning

    main
    Enhance RAG systems to handle multimedia data (PDFs, PPTs, etc.) by captioning the multimedia content and storing it alongside text data in a vector store. This allows for simultaneous retrieval of both text and multimedia assets.

  9. Implement Context Enrichment (Window around Chunk)

    main

    Context Enrichment enhances retrieval accuracy by embedding individual sentences but extending the context provided to the LLM to include neighboring sentences.

    When a relevant sentence is retrieved, the system also accesses the sentences immediately before and after it in the original text to provide a broader context window. This can be implemented using LangChain or LlamaIndex.

  10. Implement Dartboard Retrieval

    main
    Dartboard Retrieval optimizes for 'Relevant Information Gain' by combining both relevance and diversity into a single scoring function. This technique is particularly effective in dense databases where standard RAG might underperform.

  11. Implement Hierarchical Indices

    main

    Hierarchical Indices create a multi-tiered system for efficient navigation. A common implementation is a two-tiered system consisting of:

    1. Document Summaries: High-level summaries for broad navigation.
    2. Detailed Chunks: Granular text segments.

    Both tiers should contain metadata that points to the same underlying data location.

  12. Implement Multi-faceted Filtering

    main

    Multi-faceted Filtering refines retrieved results using various criteria:

    • Metadata Filtering: Filter by attributes like date, source, author, or document type.
    • Similarity Thresholds: Set minimum relevance scores to discard low-quality results.
    • Content Filtering: Remove results that fail to match specific keywords or essential criteria.
    • Diversity Filtering: Filter out near-duplicate entries to ensure a diverse set of results.