DOCSEARCHbyAlgolia
Submit Docs

typeagent Python Library

repository·main·Indexed 21 days ago

https://github.com/microsoft/typeagent-py

An experimental Python prototype for structured Retrieval-Augmented Generation (RAG) focused on incremental message indexing and querying pipelines. It includes tools for ingesting WebVTT transcripts and email data from Gmail, Outlook, and Mbox files into SQLite databases for analysis.

Tokens
38.6K
Snippets
91
Records
145
Agent score
74%

What's inside typeagent

  1. Understand TypeAgent Storage: Collections and Indexes

    main

    The Storage Layer uses two primary abstractions to manage knowledge: Collections for data storage and Indexes for efficient retrieval.

    Collections

    • MessageCollection: Stores conversation messages with ordinal indexing.
    • SemanticRefCollection: Stores semantic references and knowledge artifacts.

    Indexes

    TypeAgent uses six specialized indexes to support different query patterns:

    • SemanticRefIndex: Maps terms to SemanticRef for content discovery.
    • PropertyIndex: Maps property names to SemanticRef for structured queries.
    • TimestampToTextRangeIndex: Enables temporal navigation and filtering.
    • MessageTextIndex: Provides embedding-based semantic similarity search.
    • RelatedTermsIndex: Handles term expansion and alias resolution.
    • ConversationThreads: Organizes threads and groups context.

    Storage Providers

    You can choose between two providers that implement the same API:

    • MemoryStorageProvider: In-memory storage for fast access; data is lost when the process ends.
    • SqliteStorageProvider: SQLite-backed storage for persistent data.

  3. How message chunk loading works

    main

    Messages in the storage provider use a hybrid storage strategy for content chunks. Depending on the implementation, chunks are either stored inline or referenced externally via a URI. This allows for efficient handling of both small and large message contents.

    • Inline Storage: The chunks field contains a JSON array of strings, and chunk_uri is NULL.
    • External Storage: The chunk_uri contains a URI, and the chunks field is NULL.
    • Lazy Loading: When using external storage, the get_chunks() method loads the data from the URI on demand and caches it in memory.
    class Message:
    def __init__(self, chunks: list[str] | None = None, chunk_uri: str | None = None, ...):
        self._chunks = chunks
        self._chunk_uri = chunk_uri
        # Exactly one of chunks, chunk_uri must be not-NULL

    async def get_chunks(self) -> list[str]:
        if self._chunks is not None:
            return self._chunks
        assert self._chunk_uri is not None
        # Load chunks using chunk_uri (extractor-specific implementation)
        self._chunks = await self._load_chunks_from_uri(self._chunk_uri)
        return self._chunks

  4. Manage SQLite storage performance and reliability

    main

    When working with the SQLite implementation, keep the following performance and reliability patterns in mind:

    Performance Optimization

    • Transaction Management: Use transactions for batch operations to ensure atomicity. Consider enabling WAL mode (Write-Ahead Logging) for improved concurrency.
    • Querying: The system uses prepared statements for common queries. For large-scale workloads, consider connection pooling.
    • Memory Management: The system implements chunk caching with LRU (Least Recently Used) eviction to manage memory when using lazy loading.

    Error Handling and Integrity

    • Database Errors: The implementation handles connection failures with retry logic for transient errors.
    • Data Integrity: Foreign key constraints are used to maintain relationships and ensure referential integrity.
    • Schema Versioning: The ConversationMetadata table stores the schema version to manage migrations and ensure compatibility.

  5. Understand the SQLite Storage Provider design

    main

    The SQLite Storage Provider is designed to move typeagent from an in-memory storage model to a persistent one. It is built on three key principles:

    1. Schema Alignment: Database tables map closely to in-memory data structures. Composite objects are decomposed into separate columns, and optional fields are normalized with sensible defaults.
    2. In-Memory Compatibility: The provider is designed to coexist with the existing pure in-memory storage options.
    3. Async-First: All operations are asynchronous to support both SQLite and potential future distributed storage backends.

  6. Understand the SQLite storage implementation plan

    main

    The project is transitioning from in-memory storage to a persistent SQLite-based storage system. This migration introduces persistent indexes (term-to-semantic-ref, property, and timestamp indexes) and supports lazy loading of message chunks via chunk_uri.

    Key architectural shifts include:

    • Persistence: Moving from volatile memory to SQLite tables.
    • Indexing: Using SQLite-backed indexes for faster semantic and property-based searches.
    • Lazy Loading: The Message interface is being updated to support chunk_uri, allowing SqliteMessageCollection to load large message contents only when needed, reducing memory overhead.

  8. Configure Knowledge Extraction modes

    main

    The ContentExtractor engine supports multiple modes to balance processing speed against knowledge quality. You can choose the mode based on your requirements for structure and intelligence:

    • Basic Mode: Uses rule-based extraction focusing on titles, headings, and metadata. Best for speed and low-resource environments.
    • AI Mode: Uses LLM-powered extraction for entities, topics, and relationships. Best for high-quality structured knowledge.
    • Hybrid Enhancement: Combines rule-based extraction with AI-driven summarization.
    • Batch Processing: Allows processing multiple content items with built-in progress tracking.

    Key components involved in this process include the AIModelManager (for LLM integration and fallbacks) and the KnowledgeTranslator (which converts natural language into structured knowledge schemas).

  9. How TypeAgent Knowledge Processing Indexes work

    main

    The knowpro system uses a multi-layered indexing approach to retrieve information from conversation data. It relies on semantic references—structured knowledge objects extracted from messages—to enable efficient searching.

    Core Components

    • Primary Index: Stores the core knowledge extracted from messages.
    • Secondary Indexes: Specialized indexes for specific search types (e.g., property, time, or fuzzy search).
    • Embedding Indexes: Vector-based indexes used to find content with similar meanings.
    • Helper Indexes: Support indexes for timestamps, properties, and related terms.

    Query Processing Workflow

    1. Query Setup: Converts raw terms into structured expressions.
    2. Index Selection: The processor selects appropriate indexes based on the query type.
    3. Parallel Lookup: Multiple indexes are searched simultaneously for performance.
    4. Result Merging: Results are combined, scored, and ranked.
    5. Post-processing: Final filtering and formatting.

  10. How the ConversationSecondaryIndexes integration works

    main

    The ConversationSecondaryIndexes class acts as the integration layer between the high-level conversation objects and the underlying storage provider.

    • Lazy Loading: Indexes are not created until they are actually requested.
    • Abstraction: Index-building functions should continue to use the conversation.secondary_indexes pattern.
    • Internal Wiring: ConversationSecondaryIndexes handles the logic of pulling the real indexes from the storage provider internally, so the consumer does not need to manage the storage provider directly for standard index-building tasks.

  11. Understand the Storage Provider Architecture

    main

    The storage system uses a layered design to manage secondary indexes for conversations. The hierarchy is as follows:

    1. Index Building Functions: High-level functions (e.g., timestampindex.py, propindex.py) that construct indexes.
    2. ConversationSecondaryIndexes: The integration layer (secindex.py) that coordinates index access.
    3. Storage Provider: The implementation layer (memorystore.py or sqlitestore.py) that manages index instances.
    4. Index Implementations: The concrete data structures (e.g., SemanticRefIndex, PropertyIndex) that store the actual mappings.

    Key Integration Pattern: When building or accessing indexes, you should use conversation.secondary_indexes. This is the intended pattern for interacting with the storage layer and is not deprecated.

  12. How the TypeAgent Query Pipeline works

    main

    The Query Pipeline is a structured RAG system designed to translate natural language into precise retrieval. It follows these stages:

    1. Language Processing: Parses user intent and extracts search terms.
    2. Index Querying: Executes parallel searches across the six specialized indexes.
    3. Score Fusion: Merges and ranks results from multiple sources (entities, topics, messages, and actions).
    4. Context Building: Prepares enriched context for the LLM.
    5. Answer Generation: Synthesizes an LLM-powered response that includes citations.

    Key Features:

    • Multi-Index Search: Queries semantic and structured indexes in parallel.
    • Fallback Strategy: If structured search fails, the system automatically falls back to raw text similarity.
    • Thread Context: Automatically applies conversation thread filtering and scoping to the search.