DOCSEARCHbyAlgolia
Submit Docs

Youtu-GraphRAG Documentation

repository·main·Indexed 22 days ago

https://github.com/tencentcloudadp/youtu-graphrag

A Graph Retrieval-Augmented Generation (GraphRAG) framework using a unified agentic paradigm to perform complex reasoning over hierarchical knowledge graphs. It features a Schema-Guided Hierarchical Knowledge Tree with four levels (Attributes, Relations, Keywords, and Communities), Agentic Retrieval with Iterative Reflection (IRCoT), and dually-perceived community detection. The framework supports deployment via Docker, Conda, or local Python environments, and offers both a Web UI and a CLI for graph construction and retrieval.

Tokens
5.4K
Snippets
6
Records
25
Agent score
77%

What's inside Youtu-GraphRAG

  1. Overview of Youtu-GraphRAG

    main

    Youtu-GraphRAG is a vertically integrated reasoning paradigm based on graph schemas. It integrates the GraphRAG framework into an agent-centric organic whole. It is designed to minimize manual intervention in graph schema configuration while enabling seamless transitions between different domains (e.g., academic papers, personal knowledge bases, or corporate knowledge bases).

    Key Application Scenarios

    • Multi-step Reasoning and Summarization: Solving complex problems that require multiple reasoning steps.
    • Knowledge-Intensive Tasks: Processing problems that rely on large amounts of structured knowledge.
    • Cross-domain Expansion: Easily adapting to various fields with minimal manual schema setup.

  2. Project Structure Overview

    main

    Understanding the directory layout helps in locating specific modules:

    • config/: Configuration system including base_config.yaml and config_loader.py.
    • models/: Core logic modules:
      • constructor/: Knowledge graph construction (e.g., kt_gen.py for hierarchical graph building).
      • retriever/: Search engines like enhanced_kt_retriever.py and agentic_decomposer.py.
      • faiss_filter.py: DualFAISS retriever.
    • utils/: Utilities for LLM calls, graph processing, and evaluation.
    • schemas/: Data structure definitions.
    • output/: Generated artifacts including graphs/, chunks/, and logs/.
    • main.py: The primary entry point for execution.
    • setup_env.sh: Script to install web service dependencies.
    • start.sh: Script to launch the web service.

  3. Core Innovations of Youtu-GraphRAG

    main

    Youtu-GraphRAG introduces several key architectural innovations:

    1. Hierarchical Knowledge Tree Construction via Schema

    • Seed Graph Schema: Provides precise constraints for automatic extraction by introducing targeted entity types, relationship types, and attribute types.
    • Dynamic Schema Scalability: Allows schemas to expand dynamically for autonomous knowledge evolution.
    • 4-Layer Architecture:
      • Layer 1 (Attribute Layer): Stores entity attribute information.
      • Layer 2 (Relationship Layer): Builds triples representing relationships between entities.
      • Layer 3 (Keyword Layer): Establishes a keyword indexing system.
      • Layer 4 (Community Layer): Forms a hierarchical community structure.

    2. Dual Structure-Semantic Community Detection

    • Integrates structural topology characteristics with subgraph semantic information to build a comprehensive knowledge organization system. This supports both top-down filtering and bottom-up reasoning.

    3. Intelligent Iterative Retrieval

    • Schema-Aware Problem Decomposition: Decomposes complex queries into parallelizable sub-queries based on the graph schema.
    • Iterative Reasoning Mechanism: Uses an iterative search process to achieve deeper reasoning.

    4. Unified Configuration Management

    • Centralized Parameter Management: All components can be configured via a single YAML file.
    • Runtime Dynamic Adjustment: Supports modifying configuration parameters dynamically during program execution.

  4. Understand the Youtu-GraphRAG Four-Layer Knowledge Tree

    main

    Youtu-GraphRAG builds a hierarchical knowledge tree based on a provided Schema. The structure consists of four layers:

    1. Attribute Layer (属性层): Stores specific attribute information for entities.
    2. Relation Layer (关系层): Constructs relationship triplets between entities.
    3. Keyword Layer (关键词层): Establishes a keyword indexing system.
    4. Community Layer (社区层): Forms a hierarchical community structure for high-level knowledge abstraction.

    Built graphs are stored in output/graphs/ and are compatible with direct import into Neo4j for visualization.

  5. Understand the Youtu-GraphRAG Framework Architecture

    main

    Youtu-GraphRAG is a vertically unified agentic paradigm for Graph Retrieval-Augmented Generation (GraphRAG). It uses a Schema-Guided Hierarchical Knowledge Tree consisting of four levels:

    1. Level 1 (Attributes): Entity property information.
    2. Level 2 (Relations): Entity relationship triples.
    3. Level 3 (Keywords): Keyword indexing.
    4. Level 4 (Communities): Hierarchical community structure.

    Key components include:

    • Schema-Guided Construction: Uses seed schemas to bound extraction agents and expands them for new domains.
    • Dually-Perceived Community Detection: Fuses structural topology with subgraph semantics.
    • Agentic Retrieval: Uses schema-aware decomposition to transform complex queries into parallel sub-queries and employs Iterative Reflection (IRCoT) for advanced reasoning.

  7. Install Youtu-GraphRAG via Docker

    main

    To run Youtu-GraphRAG in a containerized environment, follow these steps:

    1. Clone the repository.
    2. Create a .env file from .env.example and configure your LLM credentials (using OpenAI format).
    3. Build the Docker image.
    4. Run the container and map port 8000.

    LLM Configuration Requirements: Your .env file must include:

    • LLM_MODEL: e.g., deepseek-chat
    • LLM_BASE_URL: e.g., https://api.deepseek.com
    • LLM_API_KEY: Your API key

    Once running, you can access the web interface at http://localhost:8000.

    # 1. Clone Youtu-GraphRAG project
git clone https://github.com/TencentCloudADP/youtu-graphrag

# 2. Create .env according to .env.example
cd youtu-graphrag && cp .env.example .env
# Config your LLM api in .env as OpenAI API format
# LLM_MODEL=deepseek-chat
# LLM_BASE_URL=https://api.deepseek.com
# LLM_API_KEY=sk-xxxxxx

# 3. Build with dockerfile 
docker build -t youtu_graphrag:v1 .

# 4. Docker run
docker run -d -p 8000:8000 youtu_graphrag:v1

# 5. Visit http://localhost:8000
curl -v http://localhost:8000

  8. Quick Start: Conda Environment Setup

    main

    For local development or batch processing without Docker, use Conda to manage the Python environment.

    1. Clone the repository.
    2. Configure .env with your LLM credentials.
    3. Create and activate a Conda environment (Python 3.10 recommended).
    4. Install dependencies using the provided setup_env.sh script.
    5. Start the Web service using start.sh.

  9. Deploy Youtu-GraphRAG via Source Code

    main

    To run the service directly using Python, ensure you have Python 3.10 installed.

    1. Clone the repository.
    2. Create a .env file and configure your LLM API credentials.
    3. Create and activate a Python virtual environment.
    4. Run the environment setup script ./setup_env.sh.
    5. Start the service using ./start.sh.
    6. Access the service at http://localhost:8000.

  10. Install Youtu-GraphRAG via Conda

    main

    For local development using Conda, follow these steps:

    1. Clone the repository.
    2. Create and configure a .env file with your LLM credentials.
    3. Create a Python 3.10 Conda environment.
    4. Run the provided setup_env.sh script to install dependencies.
    5. Start the web server using start.sh.
    # 1. Clone Youtu-GraphRAG project
git clone https://github.com/TencentCloudADP/youtu-graphrag

# 2. Create .env according to .env.example
cd youtu-graphrag && cp .env.example .env
# Config your LLM api in .env as OpenAI API format
LLM_MODEL=deepseek-chat
LLM_BASE_URL=https://api.deepseek.com
LLM_API_KEY=sk-xxxxxx

# 3. Create the conda environment.
conda create -n YouTuGraphRAG python=3.10
conda activate YouTuGraphRAG

# 4. Setup environment
chmod +x setup_env.sh
./setup_env.sh

# 5. Start the web server (for web interface)
chmod +x start.sh
./start.sh

  12. Deploy Youtu-GraphRAG using Docker

    main

    The recommended way to run Youtu-GraphRAG is via Docker to avoid environment discrepancies.

    1. Clone the repository.
    2. Create a .env file from .env.example and configure your LLM API credentials (must be compatible with the OpenAI API format).
    3. Build the Docker image.
    4. Run the container.
    5. Access the service at http://localhost:8000.