DiffMem: Differential Memory Backend

DiffMem is a FastAPI service that provides git-native persistent memory for AI agents. Instead of using a vector database or embeddings, it uses a bare git repository where each user is assigned an isolated orphan branch (user/{id}). Memory is stored as Markdown files, and history is maintained through the git commit graph.

Key Characteristics:

• Storage: Uses git worktrees mounted on demand for each user.
• Data Format: Memory is stored in Markdown with YAML frontmatter.
• No Vector DB: Relies on git history and structured Markdown for context retrieval.
• Isolation: Each user has a dedicated branch and worktree.

DiffMem is a lightweight, git-based memory backend for AI agents and conversational systems. Unlike traditional vector databases, it uses Markdown files for human-readable storage and Git to track the temporal evolution of knowledge through differentials.

Key characteristics:

• Current-State Focus: Stores only the "now" view of information in Markdown files to keep LLM context windows lean.
• Differential Intelligence: Uses Git diffs and logs to allow agents to query how facts or relationships have changed over time.
• Git-Native Retrieval: A retrieval agent uses shell commands (grep, git log, git diff, git blame) to explore the repository and build targeted context.
• Isolation: Each user is assigned an isolated orphan branch (user/{user_id}) within a single local storage repository, checked out into a per-user worktree when active.

An ontology in DiffMem is a named profile that defines the entity taxonomy for a memory repository. It specifies what kinds of entities exist, where their files are stored, and the vocabulary agents use for creating and retrieving memories.

Selecting an ontology is a deployment-time decision made via the DIFFMEM_ONTOLOGY environment variable. This setting applies to all users on that specific deployment.

The /data volume is the source of truth for DiffMem. To protect your data, use one of the following layers:

1. Hetzner Volume Snapshot (Recommended)

Provides full, binary-safe backups of the entire /data volume.

• How to enable: Use the Hetzner Cloud console to attach a Block Storage volume to your VPS, mount it at /data, and configure periodic snapshots.

2. GitHub Backup Backend

Provides an offsite mirror of user git branches. This survives VPS failure and allows for a 'cold-start restore' on a fresh VPS without manual intervention.

• How to enable: Set BACKUP_BACKEND=github and provide the necessary GITHUB_TOKEN and GITHUB_REPO_URL.
• Limitation: This only backs up git-based data; it does not backup binary or non-git data.

The WriterAgent follows a specific information flow to transform a session_transcript into structured memory:

1. Identify entities: Extract relevant entities from the raw text.
2. Create new entity files: Uses parallel LLM calls to generate files for new entities.
3. Update existing entity files: Uses parallel LLM calls to modify existing entity files.
4. Create timeline entry: Generates a new entry based on the git diff.
5. Rebuild entity semantic indexes: Rebuilds the semantic_index (a JSON descriptor appended to each entity file) using parallel LLM calls.
6. Rebuild master index: Updates the master_index.md at the worktree root, which contains all entities sorted by memory_strength.

Do not treat LLM prompts as literal instructions to be followed (e.g., asking for a loop). Instead, treat them as a way to guide the model toward specific latent spaces by including relevant concepts, keys, or tokens. Including specific terminology or 'memeplex' keys in the context attracts the model's response toward the desired behavior and fidelity.

[SalesSuprtTech]: 1.[SoltionSl]: 1a.NeedsAssessment 1b.ProposalDevelopment 1c.Negotiation 2.[TchncalSalesProc]: 2a.ProductDemo 2b.CustomSolutionDesign 2c.PostSaleSupport

The consolidation pipeline is an out-of-band repair process used to clean up, deduplicate, and link memory entities. It uses a ConsolidatorLock (via .diffmem/consolidator.lock) to prevent concurrent runs.

The Canonical Order of Operations: When chaining consolidation tools, they should be run in this order:

1. reabsorb(): (Migration only) Folds legacy entities/commitments into the owner ## Open Items block. This is opt-in and not part of the default run set.
2. dedupe(): Removes duplicate information.
3. redistribute(): Reorganizes entities into their correct locations.
4. link(): Creates semantic links between related entities.

All consolidation operations produce git commits prefixed with consolidate(...) and trigger a background backup.

The Consolidator Agent provides an out-of-band repair capability for a user's memory worktree. It is designed to fix three common failure modes that occur during high-scale session writing: duplicate entities, overstuffed user entities (catch-all files), and a lack of interlinking between memories.

To prevent slowing down the primary writer agent, consolidation is intended to be run as a separate process (either manually or on a schedule) rather than being part of the writer's hot path. It uses Git to record changes, prefixing all commit messages with consolidate: so they can be easily identified and weighted differently by retrieval agents.

Every entity file contains a ## SEMANTIC INDEX section appended automatically by DiffMem after each session.

Important: Do not edit this section manually. It is a machine-readable JSON descriptor used by the retrieval agent for fast entity triage.

Example format:

{"name":"alice_smith","type":"human","role":"Lead Engineer","strength":"Medium","hard_cues":["lead engineer","alice","smith"],"soft_cues":["direct communicator"],"emotional_cues":["trusted colleague"],"related_entities":["diff_mem","decide_git_storage"]}

DiffMem operates using three primary components:

1. Writer Agent (writer_agent): Analyzes conversation transcripts, identifies or creates entities, and stages updates in Git's working tree via explicit, atomic commits.
2. Retrieval Agent (retrieval_agent): A multi-turn LLM agent that uses a run(command="...") tool to explore the memory repository via sandboxed shell commands. It reads index.md and probes git history to output a structured retrieval plan (file sections, diffs, logs) that is resolved into context.
3. API Layer: Provides HTTP endpoints for onboarding users, processing sessions, and retrieving context. It is also available as a Python library.

Understanding these terms is essential for working with the WriterAgent:

• session_transcript: The raw text of a user-agent conversation.
• staged_changes: Modifications present in the git working directory that have not yet been committed.
• semantic_index: A JSON descriptor appended to each entity file used for fast retrieval triage.
• memory_strength: A score derived from git edit frequency and recency; this value determines the sorting order in index.md.
• master_index: The index.md file located at the worktree root, containing all entities sorted by memory_strength.
• OntologyProfile: A configuration object resolved at WriterAgent.__init__ time that drives folder routing, entity file paths, and prompt resolution.