DOCSEARCHbyAlgolia
Submit Docs

ClawMem

repository·main·Indexed 41 days ago

https://github.com/yoloshii/clawmem

An on-device, local-first memory layer for AI agents such as Claude Code, OpenClaw, and Hermes. Version 0.37.0 provides retrieval-augmented generation (RAG) using a unified SQLite vault, combining BM25, vector search, and cross-encoder reranking. It features an MCP server, lifecycle hooks for automatic context injection, and tools for importing conversation data via `clawmem mine`.

Tokens
116.4K
Snippets
307
Records
444
Agent score
50%

What's inside ClawMem

  1. Overview of ClawMem

    main

    ClawMem is an on-device memory layer designed for AI coding agents like Claude Code, OpenClaw, and Hermes. It provides retrieval-augmented search, hooks, and an MCP (Model Context Protocol) server within a single local system.

    Key characteristics:

    • Local-first: No API keys or cloud dependencies required. Runs fully on-device.
    • Hybrid Retrieval: Combines BM25, vector search, reciprocal rank fusion, query expansion, and cross-encoder reranking.
    • Unified Vault: All integration paths (Claude Code hooks, MCP server, OpenClaw plugin, or Hermes MemoryProvider) write to the same local SQLite vault, allowing different agents to share the same project memory immediately.
    • Advanced Memory Management: Features intent classification, multi-graph traversal (semantic, temporal, causal), and self-evolving metadata (keywords, tags, causal links).

  2. What is ClawMem and how does it work?

    main

    ClawMem is an open-source, on-device memory engine designed for Claude Code, OpenClaw, and other AI agents. It provides persistent, searchable memory that survives across sessions and runtime boundaries by indexing markdown documents into a local SQLite vault.

    Core Capabilities

    • Indexing: Converts markdown into a local SQLite vault using full-text search (BM25) and vector embeddings.
    • Retrieval: Automatically injects context via hooks or provides on-demand access via MCP tools or a REST API.
    • Scoring: Uses composite scoring (relevance, recency, confidence, quality, co-activation) to rank results.
    • Tracking: Maintains session history, decisions, and handoffs.
    • Graph Traversal: Uses causal and semantic graphs to answer complex questions about the history of decisions.

    Architecture Overview

    ClawMem operates through three primary interfaces:

    1. Hooks: Automatic context surfacing (e.g., context-surfacing, decision-extractor) that handles ~90% of retrieval.
    2. MCP Tools: Agent-initiated tools (e.g., memory_retrieve, query, find_similar) used for ~10% of retrieval.
    3. REST API: An optional interface for non-MCP clients (e.g., web apps or custom scripts).

    All interfaces share a single SQLite Vault (running in WAL mode) which stores documents, vectors, and memory relations.

  3. Include internal _clawmem collection in MCP retrieval

    main

    Starting in v0.21.0, the system-internal _clawmem collection (containing observations, deductions, and handoffs) is excluded from MCP retrieval tools (search, vsearch, query, query_plan, memory_retrieve, and find_similar) by default.

    To include these internal documents in your results, you must explicitly:

    1. Pass includeInternal: true in the tool arguments,
    2. Or specify _clawmem in an explicit collection filter.

  4. Understand Quality Multipliers and Co-activation Boosts

    main

    Two primary multipliers affect the final composite score:

    Quality Multiplier (0.7 - 1.3)

    Calculated as 0.7 + 0.6 * qualityScore. The qualityScore (0.0 - 1.0) is derived from:

    • Document length and structure (headings, lists).
    • Presence of decision/correction keywords.
    • Frontmatter richness.

    Co-activation Boost (1.0 - 1.15)

    Calculated as 1 + min(coCount / 10, 0.15). Documents frequently surfaced together in the same session receive up to a 15% boost.

    Implementation Detail: In the context-surfacing hook, co-activation is applied after adaptive threshold filtering. This ensures that relationship boosts cannot rescue weak results into the surfaced set; they only boost results that have already passed the threshold.

  5. Use Knowledge Graphs and Entity Triples

    main

    ClawMem uses a graph structure to enable intent_search (answering "why" or "what led to" questions).

    • memory_relations: Stores typed edges between documents (e.g., contradicts, causal).
    • entity_triples: Stores structured SPO (Subject-Predicate-Object) facts. These are extracted from <triples> blocks in observations using a canonical predicate vocabulary (e.g., adopted, migrated_to, depends_on, uses).
    • Querying: Use kg_query(entity) to query structured relationships by entity name or canonical ID.

    Graph Maintenance: Most edges are created automatically during indexing or via the decision-extractor. However, the build_graphs MCP tool should be run manually after large ingestion batches to add temporal backbone edges and bulk semantic edges.

  6. Understand Deep Escalation in the 'deep' profile

    main

    When using the deep profile, ClawMem performs an escalation strategy if the initial fast-path search (BM25 + vector) completes in under 4 seconds (within the 8-second hook timeout).

    Escalation Phases:

    1. Query expansion: The LLM generates lexical and semantic variants of the prompt to find more candidates via BM25.
    2. Cross-encoder reranking: The top 15 candidates are scored using a reranker with 2000 characters of context per candidate. Results are blended (60% original / 40% reranker).

    This behavior is designed to produce results similar to the query MCP tool. On CPU-only systems, the fast path often consumes the budget, causing escalation to be skipped naturally.

  7. How Hooks vs MCP Tools work together

    main

    ClawMem uses a 90/10 split between Hooks and MCP Tools to solve the problem of agent proactivity:

    Tier 2: Hooks (Automatic)

    Hooks handle ~90% of memory tasks. They are part of the Claude Code lifecycle and run automatically on every prompt/response, regardless of agent decision. This ensures context-surfacing (injecting relevant memory into the prompt) and decision-extraction (capturing observations) happen without the agent needing to 'remember' to call a tool.

    Tier 3: MCP Tools (Agent-initiated)

    MCP tools handle the remaining ~10% of tasks that require deep, proactive search. These are tools the agent must explicitly call when the automatic hooks provide insufficient context (e.g., searching across multiple sessions or performing complex causal analysis).

    Note for non-Claude Code environments: In environments like Hermes, Gemini, or plain MCP clients, use diary_write and diary_read for manual memory management, as they do not have the automatic hook lifecycle.

  8. Understand Composite Scoring in ClawMem

    main

    Composite scoring is a mechanism that blends multiple signals to rank memory retrieval results beyond raw search relevance. It is applied in the following contexts:

    • Hook pipeline: Used for context surfacing.
    • query tool: Uses a retrieval-tuned weight profile.
    • memory_retrieve: Applied in keyword, hybrid, causal, and complex modes.
    • search and vsearch MCP tools: Applied only when recency intent is detected (e.g., queries containing "latest", "recent", "last session").

    Important Note on Direct Retrieval: For non-recency queries, MCP vsearch and memory_retrieve rank by raw vector cosine (scoreBasis: "vector-cosine"), and MCP search ranks by raw BM25 (scoreBasis: "fts-bm25"). In these raw modes, metadata like pins only break exact score ties and do not influence ranking or filtering.

    compositeScore = (w_search * searchScore + w_recency * recencyScore + w_confidence * confidenceScore)
                 * qualityMultiplier * coActivationBoost

  9. How edges are created in the ClawMem graph

    main

    Edges are generated through several automated and manual processes:

    SourceEdge typesTrigger
    A-MEM generateMemoryLinks()semantic, supporting, contradictsIndexing (new docs only). Requires embeddings.
    A-MEM inferCausalLinks()causalPost-response via decision-extractor. Links _clawmem/agent/observations/ docs.
    Beads syncBeadsIssues()causal, supporting, semanticbeads_sync MCP tool or watcher. Maps dependency types.
    buildTemporalBackbone()temporalbuild_graphs MCP tool (manual). Creation-order edges.
    buildSemanticGraph()semanticbuild_graphs MCP tool (manual). Pure cosine similarity.

  10. How Confidence Scoring Works

    main

    The confidence score (0.0 - 1.0) starts at 0.5 for new documents and is adjusted by several factors:

    • Contradiction Detection: If a new decision contradicts an old one, the old decision's confidence is lowered (requires CLAWMEM_JUDGE_* env vars).
    • Feedback Loop: Referenced notes receive confidence boosts.
    • Attention Decay: Non-durable types (handoff, progress, conversation, note, project) lose 5% confidence per week without access. Durable types (e.g., decision, research) are exempt.

  11. How adaptiveTraversal works

    main

    adaptiveTraversal() performs a multi-hop beam search to find related context. It follows these steps:

    1. Start: Begins at anchor nodes (typically top search results).
    2. Expansion: At each hop, it expands outbound edges (all types) and inbound edges (only semantic and entity).
    3. Scoring: Candidates are scored based on edge weight (confidence), cosine similarity to the query, and depth decay.
    4. Beam Selection: Keeps the top beamWidth candidates per hop.
    5. Budget: Stops after visiting a total of budget nodes.

    Traversal Asymmetry

    To respect the directional nature of causal and temporal links, traversal is asymmetric:

    • Outbound (source→target): All types (semantic, supporting, contradicts, causal, temporal).
    • Inbound (target→source): Only semantic and entity.

    Note: For backward causal reasoning ("why did B happen?"), ClawMem uses a bounded one-hop causal step rather than widening the traversal asymmetry.

  12. Monitor recall tracking and lifecycle signals

    main

    ClawMem tracks how documents are surfaced and used to manage their lifecycle (e.g., pinning or snoozing). This is handled via two main tables:

    recall_events (Append-only log)

    Records every time context-surfacing injects a document. It stores:

    • query_hash
    • search_score
    • session_id
    • turn_index

    The feedback-loop hook later updates these to indicate if the assistant actually cited the document.

    recall_stats (Derived summary)

    Recomputed by the consolidation worker, this table provides signals used for lifecycle_status:

    SignalDescription
    recall_countTotal times surfaced
    unique_queriesDistinct query contexts
    recall_daysDistinct calendar days surfaced
    diversity_scoremin(1, max(unique_queries, recall_days) / 5)
    spacing_scoreMulti-day spread (log-scaled)
    negative_countSurfaced but not referenced (noise signal)

    Lifecycle Decisions:

    • Pin candidates: High diversity + high spacing + high recall count.
    • Snooze candidates: High recall count but mostly negative_count (noise).