dsRAG Documentation

A KnowledgeBase object is the central abstraction in dsRAG. It manages the lifecycle of documents by performing chunking, embedding, and preprocessing.

Key characteristics:

• Persistence: Objects are persistent by default. The configuration required to reconstruct the object is saved as a JSON file upon creation or update.
• Input: Accepts documents as raw text, file paths, or via the add_document method.
• Output: At query time, it returns the most relevant text segments (processed via RSE) rather than just raw chunks.

Chat threads must be persisted to allow conversations to continue across multiple interactions. You must provide an implementation of the ChatThreadDB class.

dsRAG provides two built-in implementations:

• BasicChatThreadDB: Stores chat threads in a JSON file.
• SQLiteChatThreadDB: Stores chat threads in a SQLite database.

The ChatThreadDB interface handles creating threads, retrieving them, adding interactions, and managing metadata.

A KnowledgeBase is composed of several modular components that you must provide during initialization. These components include:

• Vector Databases (dsrag.database.vector.VectorDB): For storing and searching vector embeddings.
• Chunk Databases (dsrag.database.chunk.ChunkDB): For storing the actual text chunks.
• Embedding Models (dsrag.embedding.Embedding): To convert text into vector representations.
• Rerankers (dsrag.reranker.Reranker): To refine retrieval results.
• LLM Providers (dsrag.llm.LLM): To generate responses based on retrieved context.
• File Systems (dsrag.dsparse.file_parsing.file_system.FileSystem): To handle file I/O and parsing.

When a user submits a query, dsRAG executes the following retrieval pipeline:

1. Vector database search: Initial retrieval of candidate chunks based on embedding similarity.
2. Reranking: Refining the candidate list using a reranker model.
3. RSE (Relevant Segment Extraction): Post-processing the reranked results to combine related chunks into coherent segments.
4. Results: Delivering the final, contextually rich segments to the LLM or user.

dsRAG uses three key methods to achieve higher accuracy on complex queries compared to vanilla RAG:

1. Semantic Sectioning: Uses an LLM to identify semantically cohesive sections in a document (annotated with line numbers) and generates descriptive titles for them. These titles are used in contextual headers.
2. AutoContext (Contextual Chunk Headers): Prepends document-level and section-level context (including the titles from semantic sectioning) to chunks before embedding. This provides a more accurate representation of the text for retrieval.
3. Relevant Segment Extraction (RSE): A query-time post-processing step that clusters relevant chunks and combines them into longer, coherent "segments." This allows the LLM to see a full section of text (e.g., an entire financial statement) rather than being constrained by fixed-length chunks.

dsRAG enhances standard Retrieval-Augmented Generation (RAG) performance through three core architectural methods:

1. Semantic Sectioning: Uses an LLM to partition documents into cohesive sections based on semantic meaning rather than arbitrary character counts. It identifies start/end lines and generates descriptive titles for these sections.
2. AutoContext (Contextual Chunk Headers): Prepends document-level and section-level context to individual chunks before embedding. This ensures that even small chunks carry the necessary context to be accurately represented in vector space.
3. Relevant Segment Extraction (RSE): A query-time post-processing step that clusters relevant chunks and intelligently combines them into longer, coherent segments. This is designed to handle complex questions where answers are distributed across multiple chunks.

Semantic sectioning uses an LLM to identify "semantically cohesive sections" within a document.

How it works:

1. The document is annotated with line numbers.
2. An LLM (default: gpt-4o-mini) identifies starting lines for sections.
3. Sections are broken into smaller chunks if they exceed size limits.
4. The LLM generates descriptive titles for each section.

Integration Benefit: When used with a dsRAG knowledge base, these section titles are used in contextual chunk headers (via AutoContext), which improves retrieval performance for both embeddings and rerankers.

The dsRAG chat system provides a conversational interface over your knowledge bases. It automates the following lifecycle for every interaction:

1. Message History: Maintains a persistent chat thread with the history of the conversation.
2. Query Generation: Automatically generates relevant search queries based on the user's input.
3. Knowledge Base Search: Uses those queries to search your connected knowledge bases.
4. Response Generation: Generates a response that includes citations to the source materials found during the search.

To use this system, you must provide a ChatThreadDB implementation to persist the conversation history.

dsRAG uses a hierarchical logging structure based on Python's standard logging module. This allows you to control the verbosity of specific components independently. The hierarchy is organized by functional area:

• dsrag: The root logger.
• dsrag.ingestion: Document ingestion operations.
• dsrag.query: Knowledge base querying operations.
• dsrag.chat: Chat interactions.
• dsrag.dsparse: Document parsing and chunking operations.
  • dsrag.dsparse.semantic_sectioning: Semantic sectioning operations.
  • dsrag.dsparse.chunking: Document chunking operations.