DOCSEARCHbyAlgolia
Submit Docs

RAG-Anything: All-in-One Multimodal RAG System

repository·main·Indexed 12 days ago

https://github.com/hkuds/rag-anything

An all-in-one multimodal Retrieval-Augmented Generation (RAG) framework built on LightRAG. It processes complex documents containing text, images, tables, and equations using a five-stage pipeline including MinerU parsing, multimodal analysis, and knowledge graph indexing. It features Vector-Graph Fusion for hybrid retrieval and supports various query modes, including VLM-enhanced and multimodal queries.

Tokens
35.2K
Snippets
88
Records
139
Agent score
98%

What's inside RAG-Anything

  1. Overview of RAG-Anything

    main

    RAG-Anything is an all-in-one multimodal RAG (Retrieval-Augmented Generation) framework built on top of LightRAG. It is designed to handle modern documents that contain interleaved text, images, tables, equations, charts, and other multimedia content.

    Unlike traditional text-only RAG systems, RAG-Anything provides a unified pipeline for processing and querying diverse content modalities through a single interface, making it suitable for academic research, technical documentation, and financial reports.

    Key Features

    • End-to-End Multimodal Pipeline: Covers the full workflow from document ingestion and parsing to multimodal query answering.
    • Universal Document Support: Processes PDFs, Office documents, images, and various other file formats.
    • Specialized Content Analysis: Includes dedicated processors for images, tables, mathematical equations, and heterogeneous content.
    • Multimodal Knowledge Graph: Automatically extracts entities and discovers cross-modal relationships.
    • Adaptive Processing Modes: Supports flexible workflows, including MinerU-based parsing or direct multimodal content injection.
    • Direct Content List Insertion: Allows bypassing document parsing by directly inserting pre-parsed content lists from external sources.
    • Hybrid Intelligent Retrieval: Provides advanced search capabilities across both textual and multimodal content with contextual understanding.

  2. Understand the RAG-Anything multi-stage pipeline architecture

    main

    RAG-Anything uses a layered, multi-stage pipeline designed to process heterogeneous content (text, images, tables, formulas) through a unified workflow. The pipeline consists of five primary stages:

    1. Document Parsing Stage: Uses a structured extraction engine (integrating MinerU) to identify and separate text, images, tables, and formulas while maintaining semantic links. Supports PDF, Office (DOC/DOCX/PPT/PPTX/XLS/XLSX), and images.
    2. Multi-Modal Content Understanding & Processing: Employs an autonomous routing mechanism to classify content and dispatch it to specialized concurrent pipelines. It preserves the original document's hierarchical structure.
    3. Multi-Modal Analysis Engine: Deploys modality-aware units including:
      • Visual Content Analyzer: For image analysis and spatial relationship extraction.
      • Structured Data Interpreter: For systematic interpretation of tables and statistical pattern recognition.
      • Mathematical Expression Parser: For high-precision LaTeX-based parsing of formulas.
      • Extensible Modality Processor: A plugin-based framework for custom or emerging content types.
    4. Multi-Modal Knowledge Graph Indexing: Converts content into a structured semantic representation by extracting multi-modal entities, mapping cross-modal relationships, and maintaining document hierarchy via "belongs to" relationship chains.
    5. Modality-Aware Retrieval: A hybrid system combining vector similarity search with graph traversal algorithms. It uses modality-aware ranking to adjust results based on query preferences and maintains relationship consistency for context integrity.

  3. Modality-Aware Retrieval mechanisms

    main

    RAG-Anything employs a hybrid retrieval system to ensure contextually integrated information delivery:

    • Vector-Graph Fusion: Combines vector similarity search (semantic embeddings) with graph traversal algorithms (structural relationships).
    • Modality-Aware Ranking: Uses adaptive scoring to weight results based on content type relevance and query-specific modality preferences.
    • Relational Coherence Maintenance: Ensures that retrieved elements maintain their semantic and structural relationships to preserve context.

  4. How the RAG-Anything multi-stage multimodal pipeline works

    main

    RAG-Anything uses a five-stage pipeline to process diverse content modalities. The workflow follows this sequence:

    1. Document Parsing: High-fidelity extraction using MinerU and adaptive decomposition into text, visual elements, tables, and math.
    2. Content Understanding & Processing: Autonomous categorization and routing of content through concurrent pipelines (textual and multimodal) while preserving document hierarchy.
    3. Multimodal Analysis: Specialized processing via modality-aware units (Visual, Structured Data, and Mathematical Expression parsers).
    4. Knowledge Graph Indexing: Construction of a multimodal knowledge graph that extracts entities, maps cross-modal relationships, and preserves hierarchy via belongs_to relationship chains.
    5. Modality-Aware Retrieval: A hybrid system using Vector-Graph Fusion (combining vector similarity with graph traversal) and modality-aware ranking to deliver contextually coherent results.

  5. Multimodal Analysis Engine: Specialized Analyzers

    main

    The Multimodal Analysis Engine uses specialized units to process different data types:

    • Visual Content Analyzer: Uses vision models to generate context-aware captions and extract spatial/hierarchical relationships from images.
    • Structured Data Interpreter: Interprets tables and structured formats using statistical pattern recognition to identify trends and semantic dependencies.
    • Mathematical Expression Parser: Parses complex formulas with high accuracy and provides native LaTeX support.
    • Extensible Modality Handler: A plugin-based framework that allows for the dynamic integration of new modality processors and runtime configuration of pipelines.

  6. How Page-Based vs Chunk-Based Context Works

    main

    RAGAnything supports two modes for extracting surrounding context:

    • Page-Based Context (context_mode="page"): Extracts context based on document page boundaries. It uses the page_idx field from content items. This is best for documents with a clear page structure.
    • Chunk-Based Context (context_mode="chunk"): Extracts context based on the sequential position of content items in a list. This provides fine-grained control regardless of page breaks and is useful for continuous streams of content.

  7. Select a conversion backend

    main

    You can choose how the conversion is performed using the method parameter in conversion methods.

    Available Methods

    • auto: Automatically selects the best available backend.
    • weasyprint: Uses the WeasyPrint engine (Recommended for web-style layouts and CSS).
    • pandoc: Uses Pandoc (Best for academic/LaTeX-quality output).
    • pandoc_system: Uses a system-installed Pandoc.

    Use converter.get_backend_info() to inspect which backends are currently available on your system.

    converter = EnhancedMarkdownConverter()
backend_info = converter.get_backend_info()

print("Available backends:")
for backend, available in backend_info["available_backends"].items():
    status = "✅" if available else "❌"
    print(f"  {status} {backend}")

# Use a specific backend
converter.convert_file_to_pdf(
    input_path="document.md",
    output_path="document.pdf",
    method="weasyprint"
)

  8. Querying RAG-Anything: Text, VLM, and Multimodal modes

    main

    RAG-Anything supports three distinct query patterns:

    1. Pure Text Queries: Standard knowledge base search using aquery() or query(). Supported modes include hybrid, local, global, and naive.
    2. VLM Enhanced Queries: When a vision_model_func is provided, the system can automatically analyze images in the retrieved context using a Vision Language Model. You can manually control this with the vlm_enhanced boolean flag in aquery().
    3. Multimodal Queries: Use aquery_with_multimodal() to perform queries that include specific multimodal content (tables, equations, etc.) as part of the prompt context.

    Multimodal Content Formats for aquery_with_multimodal:

    • Table: {"type": "table", "table_data": "markdown_string", "table_caption": "caption"}
    • Equation: {"type": "equation", "latex": "latex_string", "equation_caption": "caption"}
    # Pure text query modes
await rag.aquery("question", mode="hybrid")
await rag.aquery("question", mode="local")
await rag.aquery("question", mode="global")
await rag.aquery("question", mode="naive")

# VLM Enhanced (Automatic or Manual)
await rag.aquery("question", mode="hybrid", vlm_enhanced=True)

# Multimodal query with table
await rag.aquery_with_multimodal(
    "Compare metrics",
    multimodal_content=[{"type": "table", "table_data": "| A | B |\n|---|---|\n| 1 | 2 |", "table_caption": "Comparison"}],
    mode="hybrid"
)

  9. Multimodal Knowledge Graph Indexing details

    main

    The knowledge graph module transforms document content into structured semantic representations using these core functions:

    • Multi-Modal Entity Extraction: Converts multimodal elements into graph entities with semantic annotations.
    • Cross-Modal Relationship Mapping: Uses automated inference to connect textual entities with multimodal components.
    • Hierarchical Structure Preservation: Uses belongs_to relationship chains to maintain the original document's logical organization.
    • Weighted Relationship Scoring: Assigns quantitative relevance scores to relationships based on semantic proximity and contextual significance.

  10. Enable debug logging for EnhancedMarkdownConverter

    main

    To troubleshoot conversion issues, enable detailed logging using Python's logging module before initializing the EnhancedMarkdownConverter.

    import logging

# Enable debug logging
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Create converter with debug logging
converter = EnhancedMarkdownConverter()
result = converter.convert_file_to_pdf("test.md", "test.pdf")

  11. Optimize vLLM performance and memory

    main

    Use the following vLLM flags to tune performance:

    • Tensor Parallelism: Distribute models across GPUs using --tensor-parallel-size <num_gpus>.
    • GPU Memory Utilization: Adjust headroom with --gpu-memory-utilization <float> (default 0.9).
    • Max Model Length: Reduce memory usage by limiting context with --max-model-len <int> (e.g., 8192 is often sufficient for RAG).
    • Quantization: Use --quantization awq or --quantization gptq to reduce memory footprint.
    • Speculative Decoding: Speed up generation using a draft model with --speculative-model <model_name> and --num-speculative-tokens <int> (requires vLLM ≥ 0.4).