DOCSEARCHbyAlgolia
Submit Docs

MuninnDB

repository·develop·Indexed 43 days ago

https://github.com/scrypster/muninndb

A cognitive memory database that uses semantic triggers and Hebbian learning to surface relevant information. It supports MCP, REST, gRPC, and SDKs for Python, Go, Node.js, and PHP. Features include a Cortex-to-Lobe replication system with Eventual, Strong, and BoundedStaleness consistency modes, and integration with AI tools like Claude Desktop, Cursor, and GitHub Copilot via the Model Context Protocol.

Tokens
123.3K
Snippets
260
Records
549
Agent score
55%

What's inside MuninnDB

  1. Overview of MuninnDB Architecture

    develop

    MuninnDB is a purpose-built cognitive database delivered as a single Go binary. It uses Pebble for raw key-value storage but manages its own storage format (ERF), indexes, cognitive workers, and activation engine. It is designed to be dependency-free at runtime.

    Interface Layer

    MuninnDB exposes several protocols for interaction:

    • MBP (Port 8474): Native binary protocol over TCP.
    • REST API (Port 8475): JSON-based interface.
    • Web UI (Port 8476): Includes health and metrics endpoints.
    • gRPC (Port 8477): Protocol buffers over TCP.
    • MCP (Port 8750): Model Context Protocol via JSON-RPC.

    Core Components

    • Plugin Layer (Optional): Includes Embed (HNSW vectors) and Enrich (LLM semantic analysis).
    • Core Engine: Handles the write path, the 6-phase ACTIVATE pipeline, cognitive workers (Hebbian, contradiction, confidence, transition), and the semantic trigger system.
    • Index Layer: Supports Inverted (BM25), HNSW (vector), Adjacency (association graph), and Secondary (state, tag, creator) indexes.
    • Storage Layer: Uses ERF (Engram Record Format) with Pebble KV, organized into Hot (in-memory), Warm (on-disk), and Cold (compressed) tiers.

  2. What is an Engram in MuninnDB?

    develop

    An Engram is the core data abstraction in MuninnDB, modeled after biological memory traces. Unlike a standard database row or document, an engram is not just a container for data; it is a living record that carries its own cognitive state.

    Key characteristics include:

    • Native Cognitive Metadata: It stores confidence, relevance, stability, and access patterns as first-class, indexed primitives.
    • Hebbian Associations: Relationships between engrams are not static. They strengthen when engrams are co-activated and weaken when they go dormant.
    • Temporal Scoring: Engrams use access frequency and recency (via ACT-R scoring) to determine their importance, meaning relevance must be 'earned' through use.
    • Non-destructive Lifecycle: Engrams can be archived or soft-deleted, but they remain part of the association graph to preserve the integrity of the memory trace.

  3. Observe HNSW search degradation in activation

    develop

    When performing activation, if the HNSW (Hierarchical Navigable Small World) vector search fails, the system is designed to gracefully degrade to the FTS (Full Text Search) path. To monitor this, check your logs for a warning from activation/engine.go.

    If HNSW fails, you will see a log entry similar to: activation: hnsw search degraded containing the vault ID and the specific err context. This allows operators to detect when vector retrieval is failing while the system continues to function via FTS.

  4. Understand MuninnDB memory and retrieval concepts

    develop

    MuninnDB is built around the concept of Engrams—the core data structure representing memories. To understand the system's mental model, explore these topics:

    • Memory Structure: Learn about the Engram lifecycle, fields, and key-space layout, as well as Cognitive Primitives like Hebbian learning, temporal decay, and Bayesian confidence.
    • Retrieval: Understand the ACTIVATE pipeline, a 6-phase process used to handle recall queries.
    • Organization: Learn how to use Hierarchical Memory (tree structures for tasks/plans) and the Entity Graph (named entity extraction and relationships).

  5. Understand MuninnDB's two-layer authentication model

    develop

    MuninnDB uses two distinct authentication layers to separate system administration from data access:

    1. Layer 1: Admin Credentials: Used for the system operator layer, including the Web UI, the muninn shell, and vault management endpoints. Admin users do not typically interact with vault data directly.
    2. Layer 2: Vault API Keys: Used to access specific vaults. Vaults can be open (no key required, like the default vault) or locked (requires an API key).

    Important Note on Reads: Unlike traditional databases, reads in MuninnDB are not transparent. Activating a memory (reading it) updates its access count, strengthens Hebbian weights, refreshes its temporal score, and shifts RRF fusion scores. This means a read-only consumer can change the cognitive state of the vault for all other users.

  6. How Temporal Priority (ACT-R) works in MuninnDB

    develop

    MuninnDB uses Temporal Priority based on the ACT-R (Adaptive Control of Thought — Rational) cognitive architecture to manage memory relevance. Instead of using a blunt Time-To-Live (TTL) that deletes old data, MuninnDB calculates a base-level activation score for every engram.

    Relevance is a function of how frequently (n) and how recently (ageDays) an engram has been accessed. This ensures that frequently used facts remain highly available, while unused facts naturally fade in importance without being deleted from storage.

    Key Characteristics:

    • Computed at query time: Activation is not stored or mutated by background workers; it is derived from the access count and last-access timestamp during the query process.
    • Total-recall design: No data is ever degraded in storage. The same engram queried at the same moment will always produce the same score.
    • Mathematical Foundation: The raw activation B is passed through a softplus function (ln(1 + e^x)) to produce a smooth, non-negative weight used for scoring.
    B(M) = ln(n + 1) - 0.5 × ln(ageDays / (n + 1))

  7. How Contradictions are Detected and Handled

    develop

    Contradictions in MuninnDB are strictly semantic and are not fabricated from simple association shapes.

    Detection Logic:

    • Matrix-based: Contradictions are identified via the ContradictionSeverity relation-matrix (e.g., Supports $\leftrightarrow$ Contradicts has a severity of 1.0).
    • Explicit Links: An explicit RelContradicts link can be created using Engine.Link.
    • No Fabrication: Linking one engram to multiple targets under the same RelType (e.g., two different references associations) does not trigger a contradiction flag or a confidence penalty.

    Status and Penalties:

    • Status Names: Contradictions are reported as declared. The confidence_penalty status can be pending or applied.
    • Confidence Penalty: When a contradiction is detected, a penalty is applied via the ConfidenceWorker.
    • Note on Legacy Data: Existing 0x0A FlagContradiction markers in older vaults are not automatically cleared. They must be resolved via the ResolveContradiction path.

  9. Avoid mixing HASH-keyed and SEQUENCE-keyed records in the same prefix

    develop

    A critical design rule in MuninnDB is that a single prefix must never mix fixed-width HASH-keyed records with fixed-width SEQUENCE-keyed records.

    Why? Range deletions (e.g., DeleteRange) performed on one type of record will inadvertently delete records of the other type if they share the same prefix and length. This was the cause of a major incident where replication log pruning deleted idempotency receipts (Issue #726).

  10. Understand entity edge reporting and live support

    develop

    Derived entity edges (relationships 0x21 and co-occurring 0x24) are only reported if they are supported by a LIVE engram. This support is determined at READ time rather than during data updates.

    Key Behaviors:

    • Monotone Ledgers: Co-occurrence (0x24) and mention (0x1F) counts are historical ledgers that only increment on writes and decrement only on hard deletes (DeleteEngram). They are not automatically decremented during soft-deletes or supersessions.
    • Read-time Filtering: To ensure accuracy, Engine.GetEntityAggregate filters edges against the set of entities currently co-mentioned by the live engrams in the response. This prevents the system from reporting edges that no longer have active support.
    • Evolve Behavior: A standard muninn_evolve carries the predecessor's entity set to the successor, maintaining live support. The discrepancy only occurs when a caller explicitly replaces the entity set during an evolve.
    • Safety Bounds: The live-support scan is capped at liveSupportScanCap (500 live engrams). If this cap is exceeded, the system abandons filtering and reports all edges (potentially including stale ones) rather than risking the deletion of real edges. A WARN will be issued.

  11. Understand MuninnDB durability tiers

    develop

    MuninnDB employs a two-tier durability model to balance high throughput with data integrity.

    1. Sync Tier (Zero Data Loss): Core data is fsynced to disk on every write. This includes engrams, scoring weights, provenance/audit trails, episodic frames, auth credentials, and schema migrations. These are considered the 'source of truth'.
    2. NoSync Tier (≤10ms Data Loss Window): Secondary data and metadata use a WAL (Write-Ahead Log) group-commit pattern. A background WAL syncer flushes these writes to disk every 10ms. This tier includes secondary indexes (associations, FTS, entity graphs), metadata (access counts, timestamps), and relevance scores.

    If a crash occurs, the NoSync tier may lose up to 10ms of recent writes, but these items are designed to be self-healing or re-derivable from the Sync tier.

  12. Understand the core concepts of MuninnDB

    develop
    MuninnDB is a cognitive memory database designed for AI agents. It manages memories, referred to as engrams, and uses a neuroscience-inspired pipeline for retrieval. The system automatically learns relationships between memories and supports multi-tenancy, multiple protocols, and clustering.