DOCSEARCHbyAlgolia
Submit Docs

Cognee AI Memory Platform

repository·main·Indexed 12 days ago

https://github.com/topoteretes/cognee

An open-source AI memory platform and library (v1.4.2) that provides AI agents with persistent long-term memory. It builds self-hosted knowledge graphs using vector embeddings, graph reasoning, and ontology generation to enrich LLM context with a semantic layer. Includes cognee-mcp, allowing the memory engine to run as a Model Context Protocol server with support for stdio, SSE, and HTTP transports.

Tokens
70.7K
Snippets
233
Records
333
Agent score
97%

What's inside Cognee

  1. What is cognee-mcp?

    main

    cognee-mcp allows you to run Cognee's memory engine as a Model Context Protocol (MCP) server. This enables AI agents to build and query memory through any MCP-compatible client, such as an IDE or terminal.

    Key features include:

    • Multiple Transports: Supports stdio (default), sse (real-time streaming), and http (recommended for web deployments).
    • Connection Modes: Supports Cloud Mode (connecting to Cognee Cloud via --serve-url or COGNEE_SERVICE_URL), API Mode (connecting to an existing Cognee FastAPI server), and local execution.
    • Minimal Memory API: Exposes core tools: remember (session cache or permanent graph memory), recall (focused querying), and forget (deletion).
    • Session-aware Memory: One remember tool handles both fast session cache and permanent graph memory.

  2. What is Cognee?

    main

    Cognee is an open-source platform designed to transform raw data into persistent and dynamic AI memory for agents. It combines vector search with graph databases to make documents semantically searchable and interconnected via relationships.

    It is designed to replace traditional RAG (Retrieval-Augmented Generation) systems with a more scalable and modular ECL (Extract, Cognify, Load) pipeline.

  3. Cognee Overview

    main
    Cognee is a memory layer for AI applications and agents. It uses an extensible, modular ECL (Extract, Cognify, Load) pipeline to build dynamic agent memory, helping to reduce hallucinations, developer workload, and costs. It allows for interconnecting and retrieving historical conversations, documents, images, and audio transcriptions.

  4. Use Coding Agents to extract developer rules

    main

    The Coding Agents module extracts developer coding rules and best practices from text (conversations, documentation, or commit messages) and associates them with their original sources in the Cognee knowledge graph.

    Key Behavior:

    • It uses LLM-powered structured extraction.
    • It is not enabled by default in the standard cognee.cognify() pipeline.
    • It runs automatically when using the cognee.memify() enrichment pipeline.

    Pipeline Position: ingestion → graph extraction → coding rule association → storage / indexing

  5. What is the ECL pipeline?

    main

    Cognee uses a modular ECL (Extract -> Cognify -> Load) pipeline to create dynamic memory for AI agents:

    • Extract: Connects to and extracts data from various sources (conversations, documents, images, audio, etc.).
    • Cognify: Transforms and structures the extracted data, often using RDF-based ontologies to improve semantic understanding.
    • Load: Loads the processed data into graph and vector databases (using Pydantic for simplified integration).

    This pipeline helps reduce hallucinations and operational costs by providing high-fidelity context to LLMs.

  6. Core Concepts: The ECL Pipeline

    main

    Cognee builds dynamic memory for AI agents using scalable and modular ECL pipelines:

    1. Extract: Pulling data from various sources (conversations, documents, images, audio transcriptions, etc.).
    2. Cognify: Transforming and organizing data into a structured format, specifically generating a knowledge graph.
    3. Load: Storing the processed data into graph and vector databases using Pydantic.

    This process allows agents to retrieve past context and reduces hallucinations by providing a structured knowledge base.

  7. How the token cost model works

    main

    The analysis compares two strategies for answering repeated queries over a fixed corpus:

    1. Full-context: queries × (corpus_tokens + query_overhead)
    2. Cognee: ingestion_tokens + queries × retrieved_context

    Key Components

    • ingestion_tokens: The one-time cost of cognee.remember() (summarizing and graph-extracting). It is calculated as multiplier × corpus_tokens, where the multiplier is the measured ingestion tokens per content token.
    • retrieved_context: The roughly constant context provided by cognee.recall() per query.
    • Reduction Milestones: The tool calculates at which query count the cumulative full-context cost becomes factor× the cumulative Cognee cost. Parity (where costs are equal) occurs at factor = 1.

    Assumptions

    • Constant retrieved context: Recall context is assumed fixed per query.
    • Embeddings excluded: Only language-model tokens are counted; embeddings are computed locally.
    • Extrapolation: Ingestion cost is scaled from sampled chunks, assuming uniform density across the corpus.

  8. Use Data-source connectors to ingest external data

    main

    Connectors allow you to pull data from external sources like Gmail, Slack, Notion, Google Drive, and Confluence into cognee memory. These are distributed as community packages under the naming convention cognee-community-connector-<source>.

    All connectors utilize the DLT ingestion subsystem, providing the following guarantees:

    • One call to ingest: Pass the connector's dlt source directly to cognee.remember(...).
    • Incremental re-sync: Using write_disposition="merge" allows for upserting by primary key, meaning re-running the ingestion only pulls the delta.
    • Forget-on-source-deletion: Records removed from the source are automatically deleted from the knowledge graph, vector, and relational stores via the orphan_cleanup path.
    • Prose ingestion: Connectors configured with dlt_utils.DOCUMENT_SOURCE_ATTR ensure that page or message text is processed through the standard cognify (LLM entity extraction) pipeline rather than just the relational schema path.

  9. Understand the Cognee examples directory structure

    main

    The examples/ directory is organized by the intent of the code provided:

    • configurations/: Backend setup, database configuration, and permissions/multi-tenancy (controlled by ENABLE_BACKEND_ACCESS_CONTROL=True).
    • custom_pipelines/: Scripts for extending the cognify process or creating complex task compositions.
    • database_examples/: Smoke tests for various supported backends.
    • demos/: Broad demonstrations of feature breadth.
    • guides/: Focused, small how-to guides for specific tasks.
    • pocs/: Research and proofs of concept (exploratory scripts where conventions evolve quickly).
    • integrations/: Data-source connectors (typically installed from cognee-community).

  10. Understand the relationship between Cache and Session Lifecycle

    main

    Cognee maintains a distinction between ephemeral session cache and persistent lifecycle records:

    1. Cache Data (e.g., PostgresCacheAdapter tables): Stores QA entries, traces, and usage logs. This data is intended to be ephemeral and typically follows a TTL (Time-To-Live) pattern (e.g., 7-day TTL).
    2. Session Lifecycle Records: Managed via cognee/modules/session_lifecycle/models.py (e.g., session_records, session_model_usage). These are persistent, alembic-managed rows used for long-term metrics and lifecycle tracking.

    Note: There is no foreign key relationship between cache rows and lifecycle rows. Cache rows expire, while lifecycle rows persist by design.

  11. How frequency weight tracking works

    main

    Cognee uses frequency weight tracking to prioritize graph elements that are frequently used in retrieval.

    • Mechanism: The apply_frequency_weights_pipeline reads QA entries from sessions and increments the frequency weights of referenced graph nodes and edges each time they are used in retrieval.
    • Weighting Logic: In the Kuzu graph adapter, frequency weights default to 1.0 (unlike feedback weights which default to 0.5). Each time an element is used, its stored frequency weight is incremented by 1.0.
    • Implementation: This is handled via the apply_frequency_weights task and managed through the apply_frequency_weights_pipeline wrapper.