DOCSEARCHbyAlgolia
Submit Docs

SAG Documentation

repository·main·Indexed 24 days ago

https://github.com/zleap-ai/sag

SAG is a retrieval architecture that combines semantic search and relational reasoning using an event-entity indexing model. It provides a complete knowledge base application for individuals and AI agents, featuring a FastAPI backend (sag-api), a standalone desktop application, and a Python package (zleap-sag). Key capabilities include knowledge ingestion, source-traced retrieval, and an agent orchestration core with MCP support.

Tokens
37.9K
Snippets
65
Records
201
Agent score
81%

What's inside SAG

  1. Overview of SAG Architecture and Capabilities

    main

    SAG (SQL-Retrieval Augmented Generation) is a retrieval architecture designed to replace traditional RAG and GraphRAG. It uses an event-entity indexing model with query-time dynamic hyperedges to provide both semantic retrieval and relational reasoning in a single system.

    Core Capabilities

    • Knowledge Ingestion: Supports file and web sources with document parsing, chunking, embedding, and event/entity extraction.
    • Search: Offers global or source-scoped retrieval using two modes:
      • Fast: Vector-based semantic retrieval.
      • Precise: Multi-mode retrieval.
    • Source Tracing: Every retrieval result and citation can be traced back to the exact original text chunk.
    • Knowledge Graph: Allows inspection of events, entities, and their associations.
    • Agent Chat: Multi-turn chat grounded in sources with clickable citations.
    • Integration: Provides a self-hosted REST/OpenAPI, OpenAI-compatible chat, MCP, and a Python package (zleap-sag).

    Data Model

    Instead of fragmented triples, SAG uses:

    • Event: Carries the complete meaning of a chunk.
    • Entity: A lightweight index and expansion point.
    • Query-time dynamic hyperedge: Created locally via SQL joins when events share entities relevant to a query.

  2. What is SAG and its core capabilities

    main

    SAG (SQL-Retrieval Augmented Generation) is a retrieval architecture designed to replace traditional RAG and GraphRAG by combining semantic retrieval and relational reasoning in a single system. It uses an event-entity index and dynamic query-time hyperedges to provide searchable, linkable, and traceable knowledge.

    Core Capabilities:

    • Knowledge Import: Automatic parsing, chunking, vectorization, and extraction of events and entities from files and web sources.
    • Retrieval: Supports both fast (vector) and precise (multi) modes across global or specific sources.
    • Source Traceability: Every retrieved result and citation maps directly back to the original text chunk.
    • Knowledge Graph: Visualizes relationships between events and entities.
    • Agent Dialogue: Multi-turn Q&A based on specific sources with clickable citations.
    • External Integration: Provides REST/OpenAPI, OpenAI-compatible interfaces, MCP (Model Context Protocol), and a Python package (zleap-sag).

  3. Choose between search and grep

    main

    Decide which retrieval method to use based on the nature of your query:

    • search: Use for natural language questions, general concepts, or fuzzy descriptions (e.g., "What is the approval chain for expenses?"). This performs semantic retrieval.
    • grep: Use for exact string matches, such as specific IDs (e.g., INV-2024), function names, or proper nouns where you need to find the precise occurrence.

  4. Handle MCP tool responses and error states

    main

    All SAG MCP tools return text content. When no information is found, the tools return Chinese placeholder strings instead of throwing errors. To implement branching logic in your application, check the prefix of the returned text for these specific empty states:

    • （无相关资料） (No relevant information)
    • （未找到该文档） (Document not found)

  5. Choose a retrieval strategy in SAG

    main

    SAG supports two primary retrieval strategies via the strategy parameter in engine.search():

    • vector (Quick/Default): Performs direct retrieval based on semantic similarity. It is faster and corresponds to the 'Quick' mode in the UI.
    • multi (Precise): Combines entity relationships with LLM re-ranking for more complete results. It corresponds to the 'Precise' mode in the UI but does not run a full independent GraphRAG.

  6. Understand the sag-api architecture layers

    main

    The sag-api service is organized into several functional layers. Understanding these helps in identifying where specific logic (like parsing, tool usage, or API routing) resides:

    • Adapter Layer (sag_api/sag/): The only place where zleap-sag is imported; handles mapping sources to the DataEngine.
    • Connector Layer (sag_api/connectors/): Manages collection abstractions and the registry (e.g., file upload to dynamic sync).
    • Document Parsing (sag_api/parsing/): Handles file conversion. Markdown is passed through; PDF uses MinerU (with automatic fallback); others are converted via MarkItDown.
    • Task Queue (sag_api/jobs/): Orchestrates background processing using an ingest $\rightarrow$ extract state machine.
    • Generation Layer (sag_api/generation/): Converts retrieval results into streaming LLM answers with citations.
    • Tool Layer (sag_api/tools/): Provides Agent tools, including built-in retrieval/entities and remote MCP adapters via a unified Tool interface.
    • Agent Core (sag_agent/): An independent orchestration core managing lifecycles, events, tools, approvals, cancellations, and storage ports.
    • Agent Adapter (sag_api/services/agent_service.py): Integrates SAG models, tools, and sessions into the Agent Core.
    • MCP (sag_api/mcp/): Treats sources as MCP. Includes a FastMCP server with Streamable-HTTP mounting at /mcp/ and a stdio entry point.
    • Domain Services (sag_api/services/): Contains pure business logic independent of FastAPI.
    • Interface (sag_api/api/v1/): HTTP routes responsible for IO, validation, and serialization.

  7. Import Knowledge and Document Processing

    main

    SAG supports importing Markdown, text, PDF, and Office documents. The processing pipeline follows these steps:

    1. Normalization: Documents are converted to Markdown.
    2. Background Processing: Chunking, vectorization, event extraction, and entity extraction are performed.

    Parsing Logic:

    • PDF: Uses MinerU if configured; otherwise, it automatically falls back to MarkItDown.
    • Office/Text: Uses MarkItDown by default.

  8. Understanding the SAG retrieval process

    main

    SAG operates through two distinct phases: offline indexing and online retrieval.

    Offline Indexing

    1. Parse documents into semantically coherent chunks.
    2. Parallelly extract one event and multiple entities from each chunk.
    3. Write chunks, events, entities, and their associations into a relational store.
    4. Write representations of chunks, events, and entities into vector and full-text indices.

    Online Retrieval

    1. Identify seed entities and events using semantic and lexical signals.
    2. Use SQL to expand seed events along shared entities, forming a local candidate space.
    3. Instantiate only the hyperedges required for the current query (avoiding global graph traversal).
    4. Select the strongest evidence from event and direct chunk candidates, deduplicate, and return the original text chunks.

  9. How SAG Retrieval Works: Offline Indexing and Online Retrieval

    main

    SAG's performance is driven by a two-stage pipeline: offline indexing and online retrieval.

    Offline Indexing (Data Preparation)

    1. Parsing: Documents are parsed into semantically coherent chunks.
    2. Extraction: One event and multiple entities are extracted from each chunk in parallel.
    3. Relational Storage: Chunks, events, entities, and their associations are persisted to a relational database.
    4. Indexing: Chunk, event, and entity representations are stored in vector and full-text indexes.

    Online Retrieval (Query Execution)

    1. Seeding: Seed entities and events are found using semantic and lexical signals.
    2. Expansion: SQL joins are performed over shared entities to expand from seed events into a local candidate space.
    3. Hyperedge Instantiation: Only hyperedges relevant to the specific query are instantiated locally (no global graph traversal required).
    4. Selection: The strongest event and direct-chunk candidates are selected, deduplicated, and returned as original evidence chunks.

  10. Import and process knowledge

    main

    SAG supports importing Markdown, text, PDF, Office, and other document formats.

    Processing Pipeline:

    1. Normalization: Documents are normalized to Markdown.
    2. Background Processing: SAG performs chunking, embedding, event extraction, and entity extraction.

    File Handling:

    • PDFs: Uses MinerU if configured; otherwise falls back to MarkItDown.
    • Office/Text: Uses MarkItDown by default.

  11. Configure SAG retrieval strategies

    main

    SAG supports two primary retrieval modes:

    StrategyUI LabelImplementation
    vectorFast (default)Direct retrieval by semantic similarity for a faster response
    multiPreciseCombines entity relationships with LLM reranking for more complete results

    Note: The multi strategy does not run a separate GraphRAG implementation.

  12. How SAG's architecture works

    main

    Unlike traditional RAG (semantic similarity) or GraphRAG (offline graph construction), SAG uses a unique data model to avoid the costs of global graph maintenance and entity merging:

    • Event: Carries the full semantic meaning of a chunk (instead of breaking it into independent triplets).
    • Entity: Used for indexing and expansion, but does not replace the event's meaning.
    • Query-time Dynamic Hyperedges: Instead of pre-building a global graph, SAG uses SQL during retrieval to connect events that share entities, creating a local structure specific to the current query.
    • Original Evidence: The output boundary is always the original chunk, ensuring citations are accurate.