DOCSEARCHbyAlgolia
Submit Docs

FlashRAG Documentation

repository·main·Indexed 25 days ago

https://github.com/ruc-nlpir/flashrag

A comprehensive Python toolkit for Retrieval Augmented Generation (RAG) research. FlashRAG provides a modular framework for assembling RAG pipelines with pre-implemented state-of-the-art algorithms, 36 benchmark datasets, and a visual UI for experimentation. It includes components for retrievers (Dense, BM25, SPLADE), rerankers, refiners, and generators, with optimizations via vLLM, FastChat, and Faiss.

Tokens
15.7K
Snippets
41
Records
75
Agent score
85%

What's inside FlashRAG

  1. Overview of FlashRAG

    main

    FlashRAG is a Python toolkit designed for the reproduction and development of Retrieval Augmented Generation (RAG) research. It provides a modular framework to assemble complex RAG pipelines using various components.

    Key Features:

    • Extensive and Customizable Framework: Modular components including retrievers, rerankers, generators, and compressors.
    • Comprehensive Benchmark Datasets: 36 pre-processed RAG benchmark datasets.
    • Pre-implemented Advanced RAG Algorithms: 16 state-of-the-art RAG algorithms with reported results.
    • Efficient Preprocessing: Scripts for corpus processing, retrieval index building, and pre-retrieval of documents.
    • Optimized Execution: Integration with tools like vLLM, FastChat (for LLM inference acceleration), and Faiss (for vector index management).
    • Easy to Use UI: A visual interface to configure RAG baselines and run evaluation scripts.

  2. Overview of FlashRAG components and pipelines

    main

    FlashRAG provides modular components for building RAG workflows:

    Components

    • Judges: e.g., SKR Judge for deciding whether to retrieve.
    • Retrievers: Dense (FAISS-based, e.g., DPR, BGE, E5), BM25 (Lucene-based), and Re-rankers (Bi-encoder or Cross-encoder).
    • Refiners: Extractors, Summarizers, LLMLingua, Selective Context, and Knowledge Graph refiners.
    • Generators: Encoder-Decoder (e.g., FiD), Decoder-only (Transformers), FastChat, and vLLM-accelerated generators.

    Pipeline Types

    • Sequential: Linear execution (Query $\rightarrow$ Retriever $\rightarrow$ Refiner $\rightarrow$ Generator).
    • Conditional: Uses a judge to choose paths based on query type.
    • Branching: Executes multiple paths in parallel and merges results (e.g., REPLUG, SuRe).
    • Looping: Iterative retrieval and generation (e.g., Self-Ask, Self-RAG, FLARE, IRCoT).

  3. Explore FlashRAG RAG-Components

    main

    FlashRAG provides a modular set of components that can be combined into custom pipelines. The available components include:

    • Judger: SKR Judger (decides whether to retrieve using the SKR method).
    • Retriever:
      • Dense Retriever (Bi-encoder models like dpr, bge, e5 using faiss).
      • BM25 Retriever (Sparse retrieval based on Lucene).
      • Bi-Encoder Reranker (Matching score via bi-Encoder).
      • Cross-Encoder Reranker (Matching score via cross-encoder).
    • Refiner:
      • Extractive Refiner (Extracts important context).
      • Abstractive Refiner (Uses seq2seq models).
      • LLMLingua Refiner (Prompt compressor).
      • SelectiveContext Refiner (Prompt compressor).
      • KG Refiner (Uses Trace method to construct a knowledge graph).
    • Generator:
      • Encoder-Decoder Generator (Supports Fusion-in-Decoder/FiD).
      • Decoder-only Generator (Native transformers).
      • FastChat Generator (Accelerated with FastChat).
      • vllm Generator (Accelerated with vllm).

  4. Explore FlashRAG Pipelines

    main

    FlashRAG implements several RAG pipeline types based on inference paths:

    • Sequential: Linear execution (e.g., Sequential Pipeline supporting refiner and reranker).
    • Conditional: Different paths for different query types (e.g., Conditional Pipeline using a judger module).
    • Branching: Parallel execution with merged responses (e.g., REPLUG Pipeline or SuRe Pipeline).
    • Loop: Iterative retrieval and generation (e.g., Iterative Pipeline, Self-Ask Pipeline, Self-RAG Pipeline, FLARE Pipeline, IRCoT Pipeline, or Reasoning Pipeline).

  5. Understand FlashRAG's supported RAG methods and settings

    main

    FlashRAG implements 23 standard RAG works and 7 reasoning-based methods. To ensure consistency across implementations, the framework uses a uniform default setting unless otherwise specified in the method's 'Specific setting' column.

    Default Uniform Settings

    • Generator: LLAMA3-8B-instruct (input length: 2048)
    • Retriever: e5-base-v2 embedding model (retrieves 5 docs per query)
    • Prompt: A consistent default template (details available in docs/original_docs/baseline_details.md).

    Note: Because of this uniform setting, results may differ from the original papers if the original authors used different models or hyperparameters.

    Method Types Supported

    • Sequential: Standard pipeline methods (e.g., Standard RAG, LongLLMLingua).
    • Branching: Methods that split paths (e.g., SuRe, REPLUG).
    • Conditional: Methods based on conditions (e.g., SKR, Adaptive-RAG).
    • Loop: Iterative methods (e.g., Self-RAG, FLARE, IRCoT).
    • Reasoning: Newer methods combining reasoning with retrieval (e.g., Search-R1, R1-Searcher, CoRAG).

  7. Chunk a document corpus using chunk_doc_corpus.py

    main

    You can split a large document corpus into smaller chunks to build indices or ensure document lengths are compatible with your model.

    Input Format: The input must be a JSONL file where each line contains an id and a contents field. The contents field should follow the "{title}\n{text}" format.

    Output Format: The resulting JSONL file will contain entries with id (new chunk ID), doc_id (the original document ID), title, and the chunked contents.

    Execution: Run the script from the scripts directory.

    cd scripts
python chunk_doc_corpus.py --input_path input.jsonl \
                          --output_path output.jsonl \
                          --chunk_by sentence \
                          --chunk_size 512

  8. Configure parameters using the Config module

    main

    FlashRAG manages experiment parameters through the Config module, which supports two primary methods: using a YAML configuration file or a Python dictionary.

    Configuration Priority: When combining methods, the priority is applied as follows:

    1. Parameter Dict (Highest priority)
    2. Config File
    3. Default Settings (Defined in flashrag/config/basic_config.yaml)

    Parameters can be accessed using dictionary-style syntax (e.g., config['parameter_name']).

    from flashrag.config import Config

# Method 1: Using a YAML config file
config = Config(config_file_path='myconfig.yaml')

# Method 2: Using a parameter dictionary
config_dict = {'generator_model': 'llama2-7B'}
config = Config(config_dict=config_dict)

# Accessing parameters
model_name = config['generator_model']

  9. Debug RAG pipelines using the failure mode checklist

    main

    If your FlashRAG experiments run but produce unexpected results, use this structured workflow to identify the root cause:

    1. Collect Failures: Select 10–20 examples where the score is lower than expected or the output is qualitatively wrong.
    2. Map to Failure Modes: Categorize each failure using the following patterns:
      • Retrieval issues (unrelated results, missing spans): Check for [IN] hallucination & chunk drift or [IN] semantic ≠ embedding.
      • Reasoning issues (correct chunks but drifting/mixed answers): Check for [RE] interpretation collapse, [RE] long reasoning chains, or [RE] logic collapse & recovery.
      • Observability issues (unclear why a document was retrieved): Check for [IN] debugging is a black box.
      • Infrastructure issues (initial runs fail due to unready indices/models): Check for [OP] bootstrap ordering or [OP] pre-deploy collapse.
    3. Inspect the Layer:
      • Retrieval failures $\rightarrow$ Inspect corpus, chunking, embedding models, and retrieval config.
      • Reasoning failures $\rightarrow$ Inspect prompts, reasoning depth, and generator consumption logic.
      • Observability failures $\rightarrow$ Inspect logging, attribution, and end-to-end trace tools.
      • Infrastructure failures $\rightarrow$ Inspect run order, index-building steps, and environment/version consistency.
    4. Iterate: Make a small change (e.g., adjusting a prompt or retriever setting), re-run on a subset, and verify if the failure mode pattern changes or disappears.

  11. Access supported datasets and indices

    main

    Datasets

    All 36 supported datasets (including NQ, TriviaQA, HotpotQA, etc.) are available on Hugging Face.

    Preprocessed Index

    A preprocessed index for the wiki18_100w dataset (created using the e5-base-v2 retriever) is available on ModelScope. This index is consistent with the experiments conducted by the FlashRAG authors.

  12. Run benchmarking experiments with FlashRAG

    main

    To replicate the benchmarking experiments, download the required models and configure the experiment settings. Use the provided run_exp.py script to execute the experiments.

    General Configuration Guidelines:

    • Retrieval: For standard experiments, use E5-base-v2. For BM25, use Pyserini. Set max_padding_length to 512 and max_query_padding_length to 128. Use a batch size of 1024 with fp16 enabled.
    • Generation: Use Llama3-8B-instruct (or Qwen1.5-14B) via the vllm framework. Set max_input_length to 2048 and max_output_length to 32.
    • Prompting: The system uses a unified prompt template. The system prompt instructs the model to answer based on provided documents, and documents are formatted as Doc N (Title:{title}) {content}.
    • Sampling: To match the benchmark results exactly, set test_sample_num to 1000 and set random_sample to false in your configuration.